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FOREWORD 


Literacy is no longer defined as just the ability to read and write in a decontex- 
tualised way. It has expanded to include the literacies required for specific school 
disciplines, trades and professions, the ability to understand and produce digital 
and non-digital texts that combine a range of modes of expression, and the ‘soft 
skills’ needed in contemporary workplaces, such as collaboration, communica- 
tion, creativity, and problem solving. 

In this book, the authors focus especially on literacies with new digital media. 
Convinced that the future of literacy will be shaped by digital technologies, they 
set out to map that complex terrain, surveying a wide range of technologies, from 
interactive graphics, animation software and videogames, to the latest virtual real- 
ity and mixed reality technologies, and an equally wide range of educational 
applications of these technologies, from teaching reading and finding informa- 
tion, to understanding and producing science infographics and story structures. 

Digital learning resources are often produced as though they are self-sufficient, 
as though learning takes place solely through the interaction between child and 
machine, so that teachers are no longer needed. I remember doing a multimodal 
analysis of an educational CD-Rom in the late 1990s. As was often the case in 
that period, the home page was pictorial. It showed a classroom with monitors, 
computers, and other equipment, and in a corner, seated behind a small table 
with a vase of flowers, a teacher. All items of equipment could be clicked on, 
leading to a range of educational journeys. The teacher was the only element in 
the picture that could not be ‘clicked on’. Being a teacher myself, it was a shock- 
ing moment. But the authors of this volume are firmly convinced that teachers 
have a fundamental role to play in helping children develop these new skills with 
digital media, and this book provides them with information and practical ideas 
for doing so. 


xiv Foreword 


Though often enthusing about the potential of digital technologies, the 
authors are aware of the risks these new media practices bring. Reading online, 
for instance, involves a loss of the multi-sensoriality of books — their smell, their 
tangible materiality, and their solid durability. Although digital technology is 
becoming more multi-sensorial, even experimenting with smell and taste, it will 
be important for teachers to do what technology companies cannot do — expose 
young learners to materials and embodied experiences and practices they might 
otherwise not encounter. 

Online reading also brings many distractions, and can lead to all too speedy, 
surface-level information processing, standing in the way of ‘deep learning’. 
Teachers can help children avoid this risk by setting tasks such as planning read- 
ing pathways and focusing attention. Finding information on the internet also 
has its risks. Algorithms can create ‘echo chambers’ where users only find what 
they already know and believe, rather than encountering other points of view. 
Teachers can help children to evaluate information sources on the basis of cred- 
ibility, purpose, authorship, and so on — in short, to develop what the authors 
call “epistemic critical literacy’. 

The first part of the book draws, to a large extent, on experimental research 
which links forms of embodiment with positive literacy outcomes — words are 
better understood and remembered when their meaning is also enacted by bodily 
movements of one kind or another (e.g. making flying gestures when learning 
the word ‘fly’), or when they are first written by hand. Later in the book, the 
authors draw on the social semiotic approach to multimodal communication to 
show, for instance, how, in the science classroom, children can learn to understand 
and produce effective connections between words and images in infographics, 
or in the English classroom, learn to understand and implement concepts such 
as ‘characterisation’, ‘plot development’, and ‘point of view’ by producing short 
animations. 

In short, the book not only provides a wealth of information on available digi- 
tal technologies and their potential for learning, it also shows teachers how they 
can help children realise this potential, how they can develop effective learning 
cycles and criteria for assessing children’s work, and how they can make effective 
use of examples in the classroom. 

As the book stresses, digital technologies continue to develop, and this can 
cause a sense of uncertainty. Questions inevitably remain, and the authors ask 
some of these, for instance, in the chapter on virtual reality: What existing skills 
will be relevant and what will become backgrounded or foregrounded, essential 
or cursory, in the identity performances today and in societies of the future? Who 
will be inducted into these hybrid assemblages of wearable media first, last, and 
never, and how often, and to what ends? What multisensory interactions, modes, 
bodies, and cultural content will be privileged in these virtual practices that 
are generated by technology developers in the entertainment and ‘edutainment’ 
industries? 
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Not all of these questions are, or even can be, answered at this point. But they 
must be asked and discussed, and this book does so admirably. It should be a great 
help in getting such questions also asked, and knowledgeably discussed, by teach- 


ers and teacher educators in Australia and abroad. 
Theo van Leeuwen, January 2022 
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INTRODUCTION: BEYOND 
EDUCATION FOR INDUSTRY 4.0 


Next-generation literacies 


The chapter conceives new possibilities for literacy education predicated on the 
transformation of a socio-technical revolution, one that has seen the advent of 
big data, dataveillance, and intense growth of artificial intelligence associated 
with Industry 4.0 (Australian Council of Learned Academies [ACOLA], 2020; 
Mills, 2019). There are developments in nanotechnologies moving toward quan- 
tum computing (Laucht et al., 2021), while organisations, governments, and 
markets look to new applications for IoT (Internet of Things), automation, 
robotics, wearable technologies, and biometrics for access control (Mills, 2019). 
Amid technological breakthroughs, there are seismic social shifts as the world 
encounters increasing global security threats, including climate change, terror- 
ism, cyberattacks, bio-threats, and global economic instability. 

While neuro-inspired technologies seek to develop implantable human- 
computer interfaces, like 3D printed bionic ears (Mannoor et al., 2013) and 
brain—machine interfaces, such as Neuralink (Musk, 2019), the shape of educa- 
tion, literacies, and textual practices will inevitably change. This chapter sets out 
a rationale for digital media literacies in education and their intrinsic connections 
to transformed minds, bodies, and texts. It explores the complementary relation- 
ships between these three key areas for the development of smart literacies in 
the volume, as the digitalisation of print and the social practices of reading and 
writing are redefined. It argues that information and communication technolo- 
gies are moving beyond mere substitution, augmentation, or even significant 
modification, to require new forms of literacy curriculum and learning that were 
previously inconceivable (Puentedura, 2003). 

Literacy in schools has conventionally focused on giving students complex, 
atomistic, and elaborate manuals on how to read and write, while in everyday 
life and interaction, students are similarly becoming literate by playing the 
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game (Gee, 2012). Children and young people are now surrounded by digital 
literacy practices of a virtual era, where multiple digital screens and devices 
intersect with books, pens, and paper, while they navigate “idiosyncratic 
blends of prohibitions around. ..bullying, privacy, safety, risk, and their ...digi- 
tal footprint” (Luke, et al., 2018, p. 252). Likewise, governments and education 
systems have been preoccupied with standardised national curricula to regulate 
what students should think and know, while in a world where knowledge can 
be instantly searched and found online — along with misrepresentation and 
‘fake news’ — students need to know how to think and act ethically. The influ- 
ence of digitalisation, algorithm-based media, and the age of machine learning 
on everyday textual practices is different from times past, at the turn of the 
century, or even a decade ago. 

Educational researchers have known since the 1980s that computers were 
changing the way we write, word process, program, and learn. Literacy theorists 
have conjectured since the mid-1990s, as increasingly more households were 
accessing the internet, that “literacy pedagogy must now account for the bur- 
geoning variety of text forms associated with information and multimedia tech- 
nologies” (New London Group, 1996, p. 61). At the same time bulletin boards 
and internet chats, with their radical interactivity and scrolling directionality, 
were changing earlier book-based notions of front-to-back, page-turning, linear 
strings of text (Burbules & Callister, 1996; see Chapter 2). 

As the 21st century began, we had not yet heard of YouTube, Facebook, 
Myspace, Twitter, and Skype, while literacy theorists came to terms with fea- 
tures of social media and Web 2.0 (O’Reilly, 2005). The internet became a 
space where everyday internet users could upload short posts of 140 characters 
or less. Anyone could now share a video and have a voice amidst the noise of 
the so-called democratised age of new media, or create multiple online personae 
curated for different audiences (Mills, 2015; Mills & Chandra, 2011). By the 
2010s, advances in mobile devices and applications on smartphones and tablets, 
and the widespread accessibility of faster internet, meant that literacy practices 
became ubiquitous, occurring anywhere, anytime (Mills, 2016). Virtual reality 
platforms also became of interest to consumer markets, providing textual expe- 
riences with immersive, 3D stereoscopic vision and motion tracking, and apps 
that overlay virtual content and information on the real world (Mills, 2022; see 
Chapter 7). 

Now with the unfolding 2020s, artificial intelligence and chatbots are 
anticipated to transform communication, while algorithm-based media, 
with their predictive analytics, raise new concerns as data-driven technol- 
ogy curates and creates online content and influences people’s behaviour. 
Machine learning algorithms that influence media consumption often cre- 
ate echo chambers through which users are fed information that is most like 
their earlier interests (Valtnen et al., 2019). The need for new kinds of criti- 
cal media literacy combined with technology knowledge is heightened, as 
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behavioural engineering that involves data mining and adaptive media keeps 
unsuspecting users addicted to apps and social media, able to sway the collec- 
tive emotions of populations, and influencing election outcomes (Modreanu, 
2017; see Chapter 11). 

This volume — Literacy for Digital Futures: Mind, Body, Text — addresses three 
vital changes to literacies in digital communication environments that play a role 
in reconfiguring: (i) new readers and the mind (Chapters 2—4); (ii) new writers 
and the body (Chapters 5—7); and (iii) the hybridity and multimodal grammars of 
texts that we interact with in digital spaces (Chapters 8—10). 

Part I on reading underscores the new cognitive or mental aspects of online 
reading including critical and epistemic dimensions, however, the use of ‘mind’ 
as a key organiser is about emphasis and distinction, without excluding the role 
of the body in reading practices. 

Likewise, foregrounding the ‘body’ in Part II of the volume on digital writing 
is not to deny the role of the mind or abstract thinking in text making. Rather, 
this section makes a central case for reappraising the role of the body in digital 
composition, based on the view that cognitive processes of language operate in 
conjunction with recurring patterns of embodied experience and action in the 
world. 

Part III of the book on ‘texts’ applies social semiotics to develop a new 
understanding of image-text relations and multimodal grammars of texts that 
are now frequently encountered in digital spaces. While the first two sections 
focus on the mind and body in the practices of reading and text production, 
the third section focuses on the textual changes and new multimodal grammars 
associated with the rapid circulation of digital texts. This section introduces 
systematic accounts of the patterns of representation and interaction that image- 
text grammars make available in narrative and non-narrative textual formats. 
It also importantly considers the relations between the makers and viewers of 
digital texts. 

Based on this triadic structure the book has three main aims. First, we 
respond to the heightened need for frontier approaches to preparing students’ 
minds and abilities to think, focusing on new research on evaluative think- 
ing and the changed materiality of digital and online reading (see Chapter 2). 
Epistemic shifts are undeniable in the context of a rapid circulation of multi- 
modal texts from varied knowledge sources on the internet. These changes are 
facilitated by digital information systems and new mobilities that underpin the 
ubiquitous practices of reading, playing, and tinkering with texts anywhere at 
any time, and which require the dexterous application of cutting-edge evalua- 
tive strategies and reasoning to read digitally across content areas, such as evalu- 
ating scientific information presented on the internet. Rarely addressed in new 
literacies debates, the book engages with the understanding of the skills that 
children and young people need to adjudicate conflicting digital and multi- 
modal reading sources on social, economic, and scientific concerns, such as 
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climate change, pandemics, and terrorism (see Chapter 3). The rise of video 
games as immersive digital texts also prompts more careful consideration of the 
critical thinking skills that successful gamers hone, and the myriad of oppor- 
tunities for advancing young peoples’ decision making. Learning to think in 
innovative ways, gamers may be better prepared for life beyond school as they 
develop essential futures-focused knowledge and skills to think in evaluative and 
creative ways (see Chapter 4). 

Second, we suggest novel understandings of literacy practices and knowledge 
in digital times as significantly contextual and embodied, tied to experiences of 
the world, feelings, internal states, and bodies (see Chapters 5—7). Digital literacy 
practices are changing in terms of their materiality, corporeality, and sensoriality, 
presenting innovative ways of communicating that deconstruct sensory hierarchies 
that have often emphasised vision over the multiple dimensions of literacy practice. 
The book attends to unexplored elements of the somatic nature of reading and 
writing, including the multisensoriality of practices that engage the non-visual 
senses: touch and movements of the feet (see Chapter 6), and even sound, olfac- 
tion, and taste (see Chapter 5) in multimodal, digital, virtual, and material spaces 
(Mills et al., 2018). The particulars of these changes were previously inconceivable 
beyond worlds of science fiction, as literacies today can involve, for example, talk- 
ing to someone remotely from a smartwatch, designing and printing 3D models to 
explain a concept, or making a mixed media sculpture with an electronic display 
to tell a message (see, for example, Friend & Mills, 2021). 

Third, we extend the knowledge of systematic resources to account for how 
textual meaning is increasingly multimodal, combining two or more modes in 
complex and related ways, accelerated by the rise of mass media, multimedia, 
and the internet. Consider, for example, that coded animations use digital 
programming languages to compose moving sprites and backgrounds, written 
words, sound effects, music, narration, moving images, and interactive digital 
elements (see Chapter 9). Since the mid-1990s, theorists such as the New 
London Group (1996) have drawn attention to the new grammatical features 
of texts with global exchanges and movements of hybrid textual forms, while 
others have pointed to the power held by those who have access to the codes of 
multiple virtual communities or neo-worlds — the symbolic, iconographic, and 
dictive flows. Cultural capital belongs to those who can encode and decode 
these global flows within a surfeit of communication platforms, media, and 
applications (Featherstone et al., 1995, p. 8), and these multimodal mean- 
ings are taking centre stage in education, recreational spaces, global markets, 
and workplaces. While multimodal grammars have been systematised for some 
text types, new hybrid grammars have emerged. Part IH, Texts and Digital 
Semiotics, in this volume underscores key digital text types that are becoming 
prominent in society, with direct application for educators to formulate next- 
generation school curricula (see Chapters 8—10). 
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What are literacies, multimodality, and modes? 


In the past, societal and technological changes were undeniably slower, and 
likewise, the literacies and knowledge that students needed were more stable 
and predictable, taught through time-honoured texts, curricula, and sources 
of knowledge. While much of schooling, including its routines and timetables, 
has remained unchanged, the unprecedented rate of global, technological, and 
societal change has called for a radical expansion of literacy practices and vari- 
ous kinds of schooling experiences. Newer literacy practices involve distributed 
and crowdsourced knowledge, new sensorial relations among bodies, minds, and 
technologies, and genre-blurred, digitally mediated textual practices that require 
the sophisticated deployment of semiotic resources across evermore complex 
confluences of modes. These issues are paramount for education systems in 
which teachers are entering a time of crisis — a time when extraordinarily less of 
the age-old curriculum and static text structures will survive in any recognisable 
form. 

To appreciate the uses, potentials, or constraints of new technologies for lit- 
eracy practices, it is important to understand the theoretical context in which 
literacy practices are conceived and researched. Literacy theorists have argued 
since the 1990s that conventional views of reading and writing involving the use 
of written words alone are not adequate to encompass the multimodal array of 
textual practices available (e.g. Green, 1995; Snyder, 1997). These literacy per- 
spectives have included, but are not limited to, theories of multiliteracies (New 
London Group, 1996), multimodality or social semiotics (Kress, 2000), New 
Literacy Studies or socio-cultural literacy perspectives (Gee, 2004; Street, 2003), 
sensory literacies perspectives (Mills, 2016), and earlier theories that saw literacy 
and technology as separate disciplines becoming merged together (Bigum & 
Green, 1993). Each of these perspectives included a move toward understanding 
‘literacy’ as pluralised ‘Literacies’, rather than a single, universal form of literacy 
(Mills, 2010). 

More recent definitions of literacies in digital contexts share an understand- 
ing that literacy practices are socially recognised or socially organised and use 
a symbol system and technologies to encode and disseminate representations 
(Mills, 2010; Scribner & Cole, 1981). While older technologies of encod- 
ing (e.g. quills, slates, pen, pencils, paper, and the printing press) reproduced 
literacy practices and texts in somewhat predictable ways, recent developments 
in digital technology — including the advent of the internet, Web 2.0, mobile 
devices, social media, and Industry 4.0 — have ushered in an era of exponen- 
tial expansion of hybrid literacies and textual features. Sometimes theorised as 
computer-mediated discourse (Herring, 2007), digital composition can be seen 
as influenced by both the medium (technological) and the social situation, and 
the medium is clearly being radically altered. 
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A focus on language as situated social practice has made a valuable con- 
tribution to the field of literacy research, particularly in terms of recognising 
the marginalised practices of speech communities, and in understanding the 
literacy practices of children and youth in informal contexts beyond schooling. 
This volume seeks to extend this view by attending to the role of embodied 
human action in conjunction with the materiality of the medium in meaning 
making, from pens to personal computers, typewriters to tablets, and web to 
wearables. Some, such as Gourlay (2015), have argued regarding the socio- 
cultural view of literacy practices that “this conceptual framing has fallen short 
in adequately theorising the role of material objects, in particularly, the material 
artefacts of literacy, such as papers, pens, keyboards, and mobile devices” (p. 485). 
Likewise, we seek to recentre the significance of embodied activity and senso- 
rimotor experiences in literacy practices, a dimension which has often seen a 
“lack of attention” in semiotic theories of multimodality (see Ehret & Hollett, 
2014, p. 429). 

Also essential to this volume is the concept of ‘multimodal literacy’ based 
on social semiotics and Halliday’s systemic functional linguistics (Halliday, 
1978), which sees language as essentially social and cultural, able to be modi- 
fied and understood by users based on the situational context. We have previ- 
ously defined multimodal literacy as “communication practices that involve two 
or more modes of meaning” a fundamental premise and focus of this volume 
(Mills & Unsworth, 2017, p. 2). Importantly, ‘modes’ are defined here follow- 
ing the views of Kress (2017) as culturally shaped resources for making mean- 
ing. Examples of modes include speech, writing, drawing, music, and gesture, 
among many others. The rise of mass media and the internet has contributed 
to the ease with which users can now read and produce, remix, and share mul- 
timodal texts that combine modes, such as written language, still and moving 
images, and music. 

The materiality of meaning making is also significant, because modes, such 
as written language, can be represented using different culturally specific media, 
such as on stone, clay, paper, canvas, computer screens, fabric, vinyl, billboards, 
shoes, toys, food packaging, and bodies (e.g. tattoos). The materiality of signs has 
an important semiotic function that intersects with modes (Mills & Unsworth, 
2017). The choice of materiality of sign-making is influenced by historical and 
cultural values and uses, and the availability or accessibility of the medium. For 
example, in the foreword of Mills’s (2016) volume, Literacy Theories for the 
Digital Age, anthropologist David Howes observes that the ancient quippu is a 
woven system of coloured cords and knots, thought to be a method for calcu- 
lating and record keeping by the Andean Peoples of South America. In other 
words, the quippu is an ancient cultural example of a multimodal text. The 
materiality of meaning making also involves the body in various ways, such as 
through the physiology of speech and singing, hearing, seeing, touching, smell- 
ing, tasting, moving, and so on (Kress, 2017). 
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Mind and materiality of reading in a digital age 


New forms of embodied cognition are exemplified when readers decode in 
digital spaces and activate a network of human sensory modalities to facilitate 
comprehension and understanding. Reading involves much more than just the 
mind and needs to be understood in relationship to personal bodies and interac- 
tions in the environment. In the rapidly escalating digital world, decoding texts 
to access knowledge is complex and success requires new literacy practices that 
build on traditional conceptions of multisensory stimulation of vision, olfactory 
cues, sensations of haptics related to touch, and aesthetic enjoyment as readers 
move to screens. The materiality of reading experiences shifts in the move from 
linear print to digital hypertext or non-linear formats with physical effects on 
how we engage the body and mind (Wolf et al., 2012). 

On-screen decoding changes experiences and increases visual sensory demands, 
while readers who enjoy the multisensory experiences ignited through smelling 
the paper, ink, and the glue used to make paper books, may miss the grassy notes 
and the tang of acids or a hint of vanilla over an underlying mustiness (Strli¢ et al., 
2009). And what of the changes to sensory stimulation of a print-based book 
through touch and proprioception that most certainly takes on a different feel 
when engaging with an iPad or smartphone interface? The myriad of challenges 
to sensorial experiences are also made visible when young children need to call 
on fine motor skills for tapping, swiping, pressing, and navigation around screens. 

The ‘spatio-temporal’ and the ‘imaginary’ thoughts that are seminal for creat- 
ing scenarios from what we read are also challenged by decreased and unstable 
materiality of digital multimodal texts. Such imagery helps readers recall facts, 
sequence content, retell the story, or retrieve informational scripts from memory. 
Accessing information on e-readers, smartphones, iPads, and tablets with intui- 
tive navigation of unstable texts challenges mapping the journey in the mind and 
monitoring personal reading pathways, potentially adding challenge to compre- 
hension of such texts (Barzillai et al., 2018). 

Semiotic resources on screens, such as layout, composition, text, image, 
graphics, and the range of colours, fonts, multimedia elements, animations, and 
interactive features add excitement and wonder, but also change the experience 
as readers need to filter information, make choices, and determine their personal 
pathway in hyperlinked spaces. And what about the continual interruptions and 
disruptions from popups, alerts, notifications, and hotspots that readers contend 
with? In this way, reading to make meaning on screens is fundamentally differ- 
ent to comprehension through print modes (Afflerbach & Cho, 2010; Leu & 
Maykel, 2016). In these digital spaces, we cannot just rely on traditional print- 
based text practices based on stable material anchors to aid memorising and 
comprehension, as digital texts are more cognitively demanding. 

The role of the mind in decoding texts in the digital age has never been 
more important. An epistemological approach to literacies underpins the view 
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of reading practices in this monograph, as we need innovative ways to consider 
evaluate reading and epistemic thinking as a form of critical digital literacy. We 
propose that beliefs shape evaluative judgements in critical literacy approaches 
and are the foundation for developing advanced critical readers. This is the view 
that reading practices on screens require sophisticated epistemic thinking — critical 
thinking skills to question, evaluate, and make judgements about sources of 
information. 

Because information is not the same as knowledge, readers need advanced 
epistemic skills to forge personal pathways as they move from print to screens, 
as they investigate topical social issues on the internet as part of learning across 
the curriculum, and as they develop into active citizens in the digital world. 
The rise of ‘fake news’ has eroded trust in information in the internet age, and 
there is an ever-increasing vulnerability of young people who are engaged with 
materials that are regulated by algorithms and artificial intelligence that produce 
biased sources (Lazer et al., 2018). What kinds of thinking skills need to be fore- 
grounded for futures-focused reading in societies of the future? 

Epistemic approaches to reading offer exciting opportunities for advancing 
young people’s critical literacy. Epistemology, derived from the Greek phrase 
‘study of knowledge’, has ignited scholars to raise questions related to thinking 
critically about texts. A growing body of research has explored epistemological 
conceptions, and the core beliefs of students and teachers (e.g. Barzilai & Chinn, 
2018; Barzilai & Weinstock, 2015). Greek philosophers, such as Plato and 
Aristotle, contemplated the sciences, philosophy, art, and politics by applying 
logic, reason, and inquiry, as they grappled with questions of right and wrong, 
and true and false, in attempts to understand the world and resolve complex 
issues. Socrates instigated exploration of ethics and questions of morality in soci- 
ety by inquiring to bring to the fore people’s beliefs and values. While Greek 
philosophers shaped modern thought in a time when digital spaces and ‘screen- 
agers’ or ‘digital natives’ were not even conceivable, such exploration of beliefs 
about knowledge is perhaps more important than ever today in our digital world. 

Epistemic approaches to decoding texts are extended in this volume in Part I, 
Mind and Materiality (Chapters 2—4), to consider implications as readers move 
from print to screens, as they take a critical stance to decode conflicting multi- 
modal media on topical social issues, and if they are going to succeed in immer- 
sive gaming spaces that demand razor-sharp decision making. As a provocation, 
the role of critical thinking for evaluating sources of information on the internet 
related to socio-scientific issues and associated tensions for readers are highlighted 
as seminal skills in the post-truth era, where there can be deep disagreements 
about knowing and knowledge (Chinn et al., 2020). 

Decoding multimodal content on the internet demands sophisticated epis- 
temic engagement but can also foster such skills. The role of video games is 
illustrated as an avenue for players to engage with critical thinking and prob- 
lem solving to develop complex cognition needed in a world fast-tracked to 
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continue embracing innovative technologies. Is video gaming perhaps a form 
of immersive digital text that affords the development of critical literacy and 
learning across the curriculum in a way that will propel students into the future 
and prepare them with 21st-century skills they will need for the workforce? In a 
digital world, teachers and students cannot avoid engagements with digital forms 
of multimodal meaning making and new experiences afforded by technology 
that fundamentally shift the skills needed for engagement with texts and digital 
semiotics. 


The body and senses in thought, language, and digital practice 


A central problem of many conventional views of language is that they focus 
on meaning as conceived primarily through abstract symbols that are divorced 
from their grounding in ordinary human experience and real-world refer- 
ents (Harnad, 1990). Likewise, much of cognitive science has emphasised the 
neural basis of language learning, while undermining the origin of all human 
knowledge — embodied activity in the world that begins from birth, and which 
continues throughout the life course. Recently, embodied cognition has sought 
to understand the scientific relations between mind and body, building compel- 
ling evidence for the role of bodily and sensorimotor activity in all aspects of 
language learning, from the development of early speech to vocabulary learning, 
and from comprehension to communication (Gibbs Jr, 2005). 

A sensory or embodiment approach to literacies underpins the view of digital 
media practices in this monograph, a theory first formalised in Mills’s (2016) 
volume, Literacy Theories for the Digital Age. This is the view that literacy and 
communication practices are established and developed using the full sensorium, 
and these practices vary across cultures, practices, and technologies (Mills et al., 
2018). A hallmark of human thought and language is that it can occur decoupled 
from immediate interaction with the environment — seemingly disembodied. 
But theorists of embodied cognition have demonstrated that much of so-called 
mental activity is in fact, body-based. This is perhaps the most powerful concept 
of embodied cognition (Wilson, 2002). 

How does this work? Humans internalise sensorimotor functions to become 
integral and automatic, operating covertly during abstract thought processes to 
simulate external events in the mind’s eye, drawing on visual, auditory, and kin- 
aesthetic imagery and memories. An example is how humans learn to count, first 
using one-to-one correspondence with material objects, such as one’s fingers or 
the dots on some dice, and later using only mental pictures of what was first expe- 
rienced bodily in the world (Wilson, 2002). This is crucial because the senses are 
integral to both the practical and abstract dimensions of thought and language. 

Human perception and sensorimotor activity are not secondary to the devel- 
opment of thought and language but drive and maintain this development. For 
example, we talk, and we gesture: linguists might say that the spoken words hold 
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greater importance, while the gestures are secondary, and only supportive in 
making meaning. Yet researchers of embodied cognition have found empirical 
evidence to support the view that speech and gesture are grounded in the same 
cognitive processes, with gestures having an active role in lexical access, help- 
ing speakers to organise spatial information into verbalised entities, and playing 
a critical role in planning and conceptualising the messages (Gibbs Jr, 2005). In 
other words, the movement of the hands as gestures helps the brain to tick, while 
lubricating the organisation and fluidity of speech. 

Consider another example of how embodied cognition underpins language. 
Children can learn to write using a pencil or using a keyboard, but research 
shows that they will have stronger knowledge of letter shape and formation by 
handwriting, because the motor movement of the writing hand inscribes letter 
perception and word identification in the mind in a way that does not occur 
when pressing keys while looking at a screen (Mangen & Balsvik, 2016). Here, 
we see that embodiment matters to literacy practices, and so also does the mate- 
riality of the technology. 

What then are the implications of the embodied basis of thought and language 
in a world in which the tangible materials or media of text production are rap- 
idly transforming, positioning text users in radically diverse ways? Not only can 
humans handwrite with a pencil and type using a keyboard, but they can now 
read and manipulate holographic images overlaid in their real world using smart 
glasses. Advertising companies take images using drones, while teens receive calls 
with a smartwatch, paint a picture in a virtual reality world, or communicate 
with hugs sent via Bluetooth and with senders and receivers wearing e-textiles 
(see https://cutecircuit.com/hugshirt/). Each form of new media positions the 
reader, author, listener, and speaker differently, providing materially diverse ways 
for the bodily performance of literacy practices. This opens up unprecedented 
and unresearched opportunities for new forms of embodied cognition — yet 
these implications have often been ignored in much digital literacy research (see, 
for some exceptions, Haas & McGrath, 2018; Mills, 2016). 


Texts and digital semiotics 


The systemic functional semiotic tradition developed by Michael Halliday and 
extended in Part HI of this volume, Texts and Digital Semiotics (Chapters 8—10), 
not only attends to systematising grammatical choices in texts, but is grounded 
in important understandings about children and humans as semiotic beings who 
make meaning through fundamentally embodied ways. Focusing on young chil- 
dren’s acts of meaning, Halliday (2006) theorised how language has a bodily 
basis. For example, he has used the illustration of babies who cannot use linguis- 
tic forms of expression, but whose whole bodies become animated when their 
mothers’ faces come into view, including rapid and excited limb and head move- 
ments, and facial expressions of diverse kinds, then when a mother’s attention is 
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taken away, the infant’s body becomes “listless and inactive” (p. 7). The sharing 
of attention between infant and mother is an exchange of material and bodily 
meaning called protoconversation. Halliday (2006) explains it this way: 


Matter is displaced, meaning is exchanged. I suppose we could use the 
general term ‘movement’ for both. My point is that in trying to understand 
early infancy we are faced with the unity of the material and the semiotic. 
I prefer to talk about matter and meaning — the material and the semiotic. 


(p. 7) 


Halliday (2006, p. 7) explains that the human infant cannot yet talk, but “body 
and brain” are being “stretched in anticipation” of spoken language use. At the 
foundation of Halliday’s systemic functional linguistics is the vital principle that 
language is motivated by recurring patterns of embodied experiences with the 
world — a view that is extended throughout the current volume in our focus on 
literacies of the mind and body. 

Importantly, Halliday (2006) argued that by the age of three, a child’s language 
development is systemic — with each language choice by a child having meaning 
by virtue of selection from a vast network of choices. These semiotic choices use 
a grammar and phonology that 


...Interfaces with the material world — the grammar impinging on the 
world of the child’s experience and interpersonal relationships...the pho- 
nology impinging on the world of the children’s own body (the signifying 
body...) via...phonetics and kinetics, and each of these interfaces has, in 
turn, its own systemic potential. 


(p. 8) 


Halliday’s (2006) point above explains an important connection between the role 
of the material world in a child’s language experience and socialisation, includ- 
ing the physicality of the body and the systemic potential of language systems, 
phonology, and grammar. There is no dichotomy between understanding the 
materiality of the body in language learning and systemic functional linguistic 
views of texts. This perspective is a useful basis for understanding the necessary 
relationship between the three main sections of the current volume on mind, 
body, and text. 

The changing materiality of texts, and the resultant expansion of semiotic 
choices in a digital age, is a key theme of this volume. In a seminal and prescient 
paper in the Harvard Educational Review, the New London Group (NLG) argued 
that literacy pedagogy in education would need to take into account the bur- 
geoning variety of text forms and representational modes that are associated with 
multimedia technologies (New London Group, 1996). They drew attention to 
not only the multiplicity of modes of meaning making, but also to the frequency 
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of their integration so that communication involves both the construction and 
interpretation of meanings that are realised through the collaborative expression 
of multiple modes of meaning making, such as linguistic, audio, visual, spatial, 
and gestural modes. All modes express meaning separately or in combination so 
that semiotics — the study of meaning — is at the core of understanding and teach- 
ing the contemporary and emerging forms of multimodal literacies for digital 
futures. 

Recognising the need for an expanded conceptualisation of literacy that 
incorporates multiple representational modes, the NLG coined the term ‘multi- 
literacies’ and emphasised a social semiotic perspective in describing the ongoing 
emergence of these multifaceted literacies. Social semiotics provides a robust, 
cohesive, inclusive, and meaning-centred basis for relating the diverse representa- 
tional modes. The NLG used the term ‘design’ to describe the forms of meaning. 
They explained that this term has a felicitous ambiguity in that it can describe 
the semiotic expression or artefact, such as the design of a mime or of a text, 
as well as the embodiment of the process of designing or redesigning meaning. 

The NLG regard all semiotic activity in creating or interpreting meaning 
as a matter of what they call Design, which involves three elements: Available 
Designs, Designing, and the Redesigned. Available Designs refer to the ‘gram- 
mars’ of various semiotic systems, such as images (Kress & van Leeuwen, 2020), 
sculpture and architecture (O’Toole, 1994, 2004), gesture and other forms of 
paralanguage (Martin and Zappavigna, 2019), and spoken and written language 
(Halliday & Matthiessen, 2014; Martin, 1992). Common to all modes of mean- 
ing making is the deep organising principle distinguishing three simultane- 
ously occurring, distinct, and inter-related dimensions of meaning: ideational, 
interpersonal, and textual (Halliday, 1978; Halliday & Hasan, 1985). Ideational 
meaning refers to the representation of events, participants, and circumstances in 
experience. Interpersonal meaning refers to the nature of relationships among 
participants. Finally, textual meaning pertains to the relative information value 
and cohesion among semiotic elements. 

Any semiotic activity, whatever representational modes are involved — whether 
the activity is interpreting or constructing meaning — is regarded as Designing 
by the NLG. There are two fundamental features of Designing: (i) It is always 
transformative and never simply reproductive of the meaning-making resources 
of Available Designs. All meaning making involves transformation of the avail- 
able resources of meaning through re-presentation and re-contextualisation. 
(ii) All such transformative processes of Designing involve the co-articulation of 
ideational, interpersonal, and textual meanings. Designing produces new con- 
structions and representations of reality and simultaneously transforms relations 
among the designer, the objects of designing, and those engaged with enacting 
or interpreting the designing. At the same time, coherence and relative emphases 
are established within and across the construction of reality, and the nature of the 
relationships among the people and other entities involved. 
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The Redesigned is the outcome through which meaning makers are trans- 
formed. Following systemic functional linguistics, the redesigned combines ide- 
ational, interpersonal, and textual meanings (Halliday & Hasan, 1985). While 
never simply a reproduction of any one Available Design, the new meanings that 
are made are certainly derived in a systematic way from the Available Designs 
in accordance with culturally recognised patterns of meaning. The concept of 
meaning as Design, with its three interrelated dimensions, captures the relation- 
ship between the dynamic processes of embodied meaning making, with the 
changing materiality of textual practices in digital literacy contexts. In terms of 
embodiment and language, theorists of the NLG, such as Cope and Kalantzis 
(2020) explain that meanings are configured by human bodies. This includes 
semiotic resources, such as gaze, facial expression, gesticulation, clothing, gait, 
posture, and demeanour. They similarly acknowledge that in digital environ- 
ments, wearable devices can play a role in communicating bodily meanings across 
time and space, often making digital recordings of these corporeally generated 
meanings (Cope & Kalantzis, 2020). 

Students need to learn how to negotiate not only what meanings are made, 
but also how they are made. To be able to do this, teachers and students need 
a ‘metalanguage’ — a language for talking about the meaning-making resources. 
The nature of the metalanguage continues to evolve with contributions steadily 
emerging from researchers in the systemic functional semiotic tradition, adapting 
and extending functional linguistic descriptions developed by Michael Halliday 
(1978, 1985, 1994) to many different representational modes beyond language 
(Kress & van Leeuwen, 2001; O’Toole, 1994, 2004). The evolving metalan- 
guage needs to be accessible to teachers and students, and flexible to address 
new communication forms in digital communication environments. Examples 
of our contribution to augmenting this kind of functional metalanguage are 
explained in our account of new forms of multimodal condensation of meaning 
in infographics (Chapter 8), coding animated narratives (Chapter 9), and of new 
forms of digital literary texts such as those experienced through different forms 
of virtual and augmented reality (Chapter 10). 


Implications for curriculum and pedagogy 


As classroom curriculum and pedagogy recalibrates and evolves to meet the 
learning needs of students for their digital futures, the role of critical thinking 
informed by epistemology approaches to decoding multimodal texts on devices 
will be seminal for meaning making and learning across the disciplines. Since the 
ancient Greeks, history has shown the need for questioning and inquiry if people 
are to be literate citizens of the world. Never has explicitly skilling young people 
with critical literacy for decoding, evaluating, and forming evidence-based per- 
spectives been more important, as young people read online and engage with 
topical information and misinformation. Educators can play a critical role in 
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preparing young minds of the future and preparing their students to meet challenges 
they face daily, as they defer to screens on devices for information that necessitate 
shifts in the mind and body due to the materiality of reading (Cho et al., 2018). 

This volume conveys some urgency in understanding what takes place in 
the mind and relationships with materiality as young people move from print 
to unstable hyperlinked digital spaces, and how educators can support readers 
to enjoy success on screens and advance comprehension. The critical role of 
educators in specifically advancing their students’ epistemic stance is highlighted. 
Drawing on strategies offered in this volume, educators can advance students’ 
sophisticated approaches to decoding knowledge related to socially relevant 
issues across the curriculum. This is particularly important as readers with an 
evaluative epistemic stance can significantly outperform students with absolutist 
or black-and-white thinking. Another implication for the classroom is the role 
of video game play as a source of epistemic engagement that requires advanced 
cognition, but also as a means of developing sophisticated problem-solving skills, 
as decisions about truth are never black and white, and the mind needs to com- 
pute probabilities that require evaluative thinking — key skills for the 21st century. 

The work seeks to attend to the corporeality of readers, writers, and text 
makers of the digital age, with a clear recognition that the sensorial and embod- 
ied dimensions of these meaning acts are not arbitrary (Mills, 2016), but play a 
vital role in structuring language, as well as abstract mental processes of the mind 
(Gibbs Jr, 2005). Throughout the book, we show how materiality and meaning 
making have an “inextricable connection” in producing texts in the classroom, 
whether of speech, writing, or digital composition. For example, theorists such 
as van Leeuwen (2017, p. 76), have illustrated this principle through the semiot- 
ics of voice quality. There is similarly a certain graphic materiality of writing, its 
substrates, and technologies of production. In each case, there are certain mean- 
ing potentials inherent in the materials that are socially and culturally selected 
over time, whether digital devices, parchment, wood, clay, or movements of 
the body, including fully articulated embodied systems of meaning, such as sign 
language (Kress, 2017). The materials that cultures and societies select to adapt as 
resources for making meaning have often “been neglected in traditional linguis- 
tics” (Kress & van Leeuwen, 2020, p. 237). 

Our attention on haptics in digital media practices has implications for the 
classroom, including the remarkable evidence that shows differences in language 
learning outcomes for those who handwrite versus typing with a keyboard. 
Certain tastes — sweet, salty, and bitter — are associated with the angularity of 
fonts (Velasco et al., 2015), while writing flow differs when smelling certain 
substances (Gonçalves et al., 2017), or when hearing certain music (Ransdell & 
Gilroy, 2001). The way in which some digital tools bifurcate the visual and 
motor spaces (e.g. where the gaze is directed and where the hands are operat- 
ing), makes writing less effective in certain ways, such as for knowledge of let- 
ter formation (Haas & McGrath, 2018). These and other implications for text 
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production in the classroom, including virtual, mixed, and augmented reality 
spaces, matter to the future of literacy learning in a world where the materiality 
of the medium clearly matters to the message. 

Likewise, literacy education for digital futures requires moving beyond the 
semiotics of logocentric texts in the classroom, and beyond a privileging of 
vision, to acknowledge the multiple modes of meaning making. This will neces- 
sitate curricular and pedagogic attention to how various modes make meanings, 
and how those meanings converge with or complement each other. To facilitate 
this, teachers and students need a metalanguage — a language for talking about 
the meaning-making resources of the various modes, and how they interact 
in multimodal texts (Forey, 2020; He & Forey, 2018; Love & Sandiford, 2016; 
Macken-Horarik et al., 2018; Unsworth & Macken-Horarik, 2015; Unsworth 
& Mills, 2020). The scope of a pedagogically accessible metalanguage will need 
to be expanded to accommodate new forms of meaning making through modes 
such as sound, movement, touch or haptics, olfaction, taste, gesture, and others, 
as literate practices are transformed in ways that were previously inconceivable. 
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PART | 
Mind and materiality 


People have been decoding symbols for thousands of years with this process reor- 
ganising our minds and expanding the way we think. Print-based semiotics have 
provided stable linear texts since the invention of Sumerian logographs based on 
simple pictures in 4000 BC, the introduction of the Phoenician alphabet in 2000 
BC, and later, the mass distribution of the written word via Gutenberg’s printing 
press. The advent of the personal computer as a mass-market electronic device 
revolutionised consumer engagement in the 1970s; however, texts were still largely 
linear until hypertexts made their appearance in computing systems as part of the 
World Wide Web in 1987. Since inception, digital devices and the web have been 
rapidly expanding modes of communication, escalating access to information, and 
fundamentally changing the nature of the mind and body in decoding texts. 
New modes of communication require new literacies, particularly as people 
engage with online search engines, hyperlinked webpages, multimodal media, 
video gaming, and immersive experiences, such as augmented reality and virtual 
reality. While new digital practices are quickly sweeping nations, readers can 
struggle to decode, evaluate information, and make meaning as digital hypertext 
environments demand more sophisticated skills than traditional linear text. As 
Leu et al. (2008) point out, new literacies of the internet and other informa- 
tion communication technologies include the skills, strategies, and dispositions 
necessary to successfully use and adapt to the rapidly changing information and 
communication technologies; however, a precise definition of these new litera- 
cies may never be possible to achieve because their most important characteristic 
is that they continually evolve. Coiro (2021) clearly highlights such tensions in 
the field over the last 30 years as scholars interested in digital reading continue 
to perpetuate a lack of conceptual clarity about such a notion, and risk oversim- 
plifying digital reading as a singular entity analogous to reading text on a screen. 
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In this section of the volume, we explore some of the new literacy practices 
emerging that challenge educators to think about how to conceptualise read- 
ing in digital spaces and, importantly, what this means for supporting readers to 
address monumental shifts in the skills they need (Braten et al., 2020). Ongoing 
inquiry and robust discourse are needed to recognise, engage with, and concep- 
tualise new literacies on a fast-tracked evolutionary trajectory. The chapters in 
this section evidence a range of reading practices in digital spaces — hyperlinked 
texts, multimodal webpages, video gaming environments — with a desire to 
stimulate innovative educational initiatives to recalibrate reading pedagogies for 
digital meaning making and shed light on the practices involved in successful 
digital reading in the 21st century. 

In Chapter 2, we consider how readers face new challenges in making mean- 
ing of texts in digital spaces that change the materiality and embodiment of their 
experience (Delgado et al., 2018). The complexities of reading in digital envi- 
ronments are considered through a fine-grained lens to illustrate and clarify the 
processes involved. Central to this discussion is recognition of the way shifts from 
print-based reading to digital forms of decoding, comprehension, and mean- 
ing making have implications for cognition and the way children read deeply, 
process, and comprehend information that they engage with. In this way, we 
consider what happens to the mind when readers shift to decreased and unstable 
materiality of reading on screens, and how reading digital hypertext content 
requires innovative educational initiatives to recalibrate reading pedagogies for 
digital meaning making. As decoding texts is complex, careful orchestration of 
cognition is needed, but human sensory modalities, such as vision, olfactory cues, 
sensations of haptics related to touch, aesthetics, and the material world are also 
crucial. This chapter makes visible the critical role of multisensory stimulation 
and possibilities for supporting reader processes as they engage across a range of 
digital spaces. 

The internet is a global networked digital space that provides readers with 
information from around the world. Information, however, is not knowledge. In 
Chapter 3, we consider why critical reading online requires sophisticated think- 
ing skills and a recalibration of how we conceptualise digital reading in such 
spaces. Controversial information abounds on the internet, particularly when 
it comes to issues related to socio-scientific issues — or socially relevant debates 
informed by science — such as climate change, health pandemics, mobile phone 
health risks, and phenomena related to issues such as genetic modification (Lunn 
et al., 2021). New theoretical approaches show how young people can engage 
with evaluative thinking to make informed decisions about such socially relevant, 
real-world problems, as they develop a sophisticated epistemic stance (knowing 
about knowledge) to apply as they read, synthesise, make inferences, and justify 
their interpretation of multiple sources online (Barzilai & Zohar, 2012). In this 
chapter, we argue this processing is a new way to think about critical literacy and 
refer to this approach as critical epistemic literacy. 
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While Chapter 3 looks at critical literacy for engaging in hyperlinked inter- 
net spaces and evaluating controversial information on the World Wide Web, in 
Chapter 4, we consider why video gaming is an important digital practice that 
also engages epistemic thinking (Scholes et al., 2022). Ilustrated are some of the 
advances in video games that provide opportunities for critical literacy as play- 
ers decode, evaluate, and make decisions in collective spaces of self-generated 
ideas. While traditional educational practices often emphasise one right answer 
to a problem (objectivist epistemology) video game play has the potential to 
advance literacies and provide opportunities for engagement in situated, active, 
problem-based learning environments. While serious games are often epistemic 
focused, we also consider how action and multiplayer virtual gaming environ- 
ments offer creative spaces for trialling alternative pathways and new innovations, 
or multiple right answers to a problem (subjectivist epistemology). In this chap- 
ter, we reconceptualise pedagogical approaches to knowledge and how video 
games can facilitate the epistemological to support learning environments that 
develop knowledge creation, while answering key questions about how to foster 
and develop new knowledge and critical literacies for the future. 

Taken together, this section highlights literacy practices that disrupt tradi- 
tional perspectives on decoding linear texts, adding scholarship on digital read- 
ing. Our provocations aim to challenge the status quo and encourage new ways 
of understanding readers immersed in exciting digital spaces that are changing 
the structures of the mind. At the heart of this section is our contribution to 
discourse about evolving practices and pedagogies for advancing literacy for 
digital futures. 
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2 


MIND AND MATERIALITY OF 
DIGITAL READING 


How digital reading is reconceptualising literacy 


The rapid evolution of new technologies has accelerated children’s engagement 
with mobile devices, fundamentally changing the nature of reading. Young peo- 
ple today are immersed in a multitude of digital spaces, such as multimedia web- 
sites, interactive graphics, e-books, digital e-readers — like Kobo and Amazon 
Kindle — and social media. These spaces provide many benefits, including rapid 
and expanded access to knowledge and information; however, there are also 
unanswered questions about decoding texts on the many digital devices avail- 
able. Engagement in digital spaces requires sophisticated skills for meaning mak- 
ing, arguably changing the traditional nature of reading and reconceptualising 
literacy. 

Along with entertainment and global connections, digital spaces offer oppor- 
tunities for accessing endless amounts of information that need to be processed 
by the reader. Readers increasingly face new challenges making meaning of texts 
in digital spaces that change the materiality and embodiment experience. In this 
way, shifts from print-based reading to digital forms of decoding, comprehen- 
sion, and meaning making involve very different material experiences in how we 
engage the body and mind. Material shifts in experiences raise concerns about 
how digital reading differs from print-based reading, and whether these differ- 
ences have implications for the way children read deeply, process, and compre- 
hend information. 

This chapter first explores how reading includes the ‘spatio-temporal’ that 
takes place in the mind during the act of reading, and the ‘imaginary’ — related 
to thoughts that are seminal for creating scenarios from what we read. In 
so doing, we identify how shifts from reading linear print to reading digital 
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hypertexts — non-linear formats — have physical effects on how we engage the 
mind. As experiences of the body also change when readers engage with digi- 
tal devices, we consider the role of multisensory stimulation through vision, 
olfactory cues, sensations of haptics related to touch, and aesthetic enjoyment 
in reading. These concepts of embodiment are further developed in relation to 
writing in Chapters 5 and 6 of this volume. 

As the chapter progresses, it highlights challenges involved in decreased and 
unstable materiality of reading on screens, and how reading digital hypertext 
content requires innovative educational initiatives to recalibrate reading pedago- 
gies for digital meaning making. These pedagogies account for the lack of mate- 
rial anchors and tendency for browsing and scanning, rather than deep reading, 
when engaged with reading online. Finally, digital reading is positioned as a 
seminal aspect of literacy that needs to be explicitly and systematically taught 
through the primary and secondary years of school to prepare young people for 
their digital futures. 


New understandings of the mind and materiality for 
digital reading 


Shifts in processes of the mind 


Reading is often described as a cognitive function that requires linguistic pro- 
cessing. It is not, however, a natural process as we were never born to read — we 
were born to see, move, speak, and think. Human beings invented reading only 
a few thousand years ago, rearranging the organisation of our minds which in 
turn expanded the ways individuals think (Wolf, 2018). Underlying the brain’s 
ability to learn to read is the capacity to make new connections among struc- 
tures and circuits originally devoted to more basic cognitive processes that have 
enjoyed a longer existence in human evolution, such as vision and spoken lan- 
guage (Canolty et al., 2007). The properties of print-based text can support this 
form of cognition due to the stability and linearity of text, as well as the layers 
of thought and composition that print represents, while invoking a reader’s full 
attention. Becoming fluent in the decoding process requires time and attention 
to the task through the activation of the spatio-temporal areas of the brain within 
distributed neural networks (Wolf, 2018). 

Today, we know quite a lot about the workings of the mind, as hemodynamic 
measures show the location (space), and electromagnetic measures show the tim- 
ing (time) of brain activity during language processing (Canolty et al., 2007). 
Spatio-temporal processing of texts takes place mainly through interactions of the 
temporal and inferior prefrontal areas of the left hemisphere, with understanding 
of a word peaking about 400 milliseconds after recognition (Canolty et al., 2007). 
The mind is also involved during the act of reading when we sequence plots and 
remember details of characters, settings, backstories, and conversations — as stories 
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are one efficient way to convert information into long term memory (Park et al., 
1996). The mind is extremely busy trying to organise and sort information while 
reading and works hard to predict what will happen next in the text. Readers 
also search to find connections between related events, thoughts, and details of 
what is read. Decoding texts on screens, however, can encourage fragmented 
reading or multitasking, which utilises a different form of attention, requiring a 
more nuanced speed of processing that may be changing our brains (Wolf, 2018). 
As part of these changes, processes for developing mental images may also be 
destabilised. 


Mental images 


The mind is involved in the thoughts that take place as we create mental images 
and scenarios from what we read. While books can contain beautiful pictures to 
tantalise the viewer, children can also learn to make mental images as they read 
a range of texts, with this process very active in experiences with narratives. 
Mental images, or mental representation, resembles the experience of objects, 
events, or scenes, but occurs without the presence of a tangible stimulus (Pinker, 
1980). Embodied representation in the form of simulation takes place as the 
mind enacts the sensory and motor (i.e. sensorimotor) experiences (Sadoski, 
2018). Eating an ice-cream cone, for instance, would normally involve all the 
sensory experiences related to seeing, hearing, tasting, smelling, and the haptic 
activities of holding the cone, licking the ice cream, chewing the cone — with 
simulation this experience can be activated to produce the same sensory repre- 
sentations in the mind (Sadoski, 2018). Because a central aspect of embodiment 
is motor activity, reading words that are associated with different parts of the 
body (e.g. lick, kick, pick) activates corresponding neurological motor systems 
(Pulvermiiller, 2005). 

In this way, mental images are a form of simulation. When a reader compre- 
hends texts, imagery and simulation are inseparable (Moulton & Kosslyn, 2009). 
Reading words that are associated with different parts of the body (e.g. licking 
an ice cream) activates corresponding neurological motor systems or what neu- 
roscientists refer to as ‘mirror neurons’ (Dickerson et al., 2017). Mental images 
and scenarios created while reading can add delight for engaged readers. In an 
Australian study of elementary school students’ reading habits, many children 
described intimate relationships with texts and a love of immersion in the sto- 
ryline (Scholes et al., 2021). As Jasper, a ten-year-old avid reader from the study 
explains in the vignette below, he particularly enjoys imagining what the char- 
acters are doing as the narrative unfolds, and he also likes to imagine himself in 
the story: 


Not too long ago I finished Deltora Quest — Series Three. They’re just 
really, really out of the ordinary, and I like that.... On the back of the first 
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series of Deltora Quest it says: Deltora is a land of monsters and magic, and 
that’s pretty much the thing with every single book Emily Rodda’s ever 
written, and I like that it’s all fantasy and strange creatures. 

For every sentence I imagine the people doing things, and I have my 
own little characters locked in my brain...if they’re boys I just imagine 
them as me — you can just create the picture in your mind. 


As Jasper constructs mental images, he draws on prior knowledge and back- 
ground experiences and connects the author’s story with a personal picture. 
This process can be enhanced for young readers through guided visualisation, as 
young people learn to create mental images rich in sensory content. This process 
supports comprehension as readers visualise the scenery, characters, plot, and 
actions as they read sentences in the text. These images include various forms 
of sensory stimulation, whether visual (characteristics of the objects), auditory 
(characteristics and intensity of sound), kinaesthetic (characteristics of movement 
and touch), gustatory (related to tastes), or olfactory (related to odours). 

Such imagery can then be helpful for recalling facts and sequencing to 
identify the beginning, middle, and end of events, and the ability to retell a 
story. How does this process change, then, when readers move to screens and 
engage with e-readers, smartphones, iPads, and tablets, and the intuitive navi- 
gation of unstable texts challenges mapping the journey in the mind? There 
can be issues keeping track of the beginning, middle, and end of digital texts 
on screens, along with struggles monitoring personal reading pathways which, 
in turn, potentially challenge the comprehension of text (Barzillai et al., 2018). 
Where a print-based book includes pages with letters, words and sentences 
fixed in place, the texts that appear on a screen are ephemeral images. A reader 
of digital text might scroll through a seamless stream of words, tap forward and 
backwards as part of the meaning-making process, use the search function to 
locate a keyword or a particular phrase, and click on a text or an image that 
accesses related content. In this way, shifts in reading in the digital domain 
are more cognitively demanding as they require flexibility in processes in the 
mind. Reading, however, involves much more than just the mind. The mind 
needs to be understood in the context of inter-relationships with the physical 
body and the surrounding environment. 


Shifts in embodied experiences 


While the mind is clearly involved in the task of meaning making when decod- 
ing texts, reading is also a highly embodied experience that activates a network 
of human sensory modalities. From an evolutionary perspective, humans evolved 
from beings whose neural resources were devoted primarily to perceptual and 
motoric processing, and whose cognitive activity consisted largely of immediate 
physical interaction with the environment (Wilson, 2002). Human cognition, 
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rather than being centralised, abstract, or distinct from peripheral input and out- 
put processes, may instead have deep roots in sensorimotor processing. Research 
under the auspices of embodied cognition illustrates how neurophysiological and 
neuropsychological processes involved in perception, motor action, and cogni- 
tion are more closely associated than previously understood (Calvo & Gomila, 
2008). Based on this understanding, decoding texts is complex and includes 
careful orchestration of cognition, affect and emotion (Kress, 2003), bringing 
together human sensory modalities and the surrounding material world. Then 
what is the role of multisensory stimulation through vision, olfactory cues, sen- 
sations of haptics related to touch, and aesthetic enjoyment in reading, and how 
does the sensoriality of reading shift when readers move to screens? 


Multisensory stimulation 


Skilled readers rely on vision and move their eyes during reading as movements 
(called saccades) bringing a line of text into foveal vision for detailed visual pro- 
cessing, with skilled readers making regressions back to material already read — 
about 15% of the time — and slower readers taking longer times, shorter saccades, 
and more regressions (Rayner, 1985). Educators often see aspiring readers who 
struggle to acquire new skills if their vision is compromised. As sight is clearly 
essential for reading, students feel they can ‘see’ more clearly when they engage 
with a print-based book, and they are also acutely aware that the visual sensory 
demands of reading on screens are harder on their eyes, with some students com- 
plaining about eyestrain (Baron, 2015). 

In this way, lighting is also a salient factor as readers are fixed to devices 
for long periods of time, with concern that blue light shining from screens 
is harmful to health. While natural blue light from the sun boosts alertness, 
helps memory and cognitive function, and elevates mood, there are concerns 
about ocular overload (retinal cell damage, vision problems, macular degen- 
eration, cataracts), with prolonged exposure to blue light on devices not in a 
natural setting (Chu et al., 2011). The role of vision in reading is highlighted 
and given prominence; however, the roles of the many other senses are also 
important. 

While you might think that smell would be rather inconsequential in rela- 
tion to the enjoyment of reading, it turns out that olfactory cues often elicit an 
emotional, nostalgic response to handling hardcopy books. Early in life, olfactory 
cues are important, and infants are attracted to maternal odours, with the smell 
of a mother’s clothes comforting to newborns (Schaal et al., 2004). In a similar 
manner, multisensory experiences are ignited for readers through smelling the 
paper, ink, and the glue used to make paper books. In a study across five coun- 
tries, Baron (2015) found that print was aesthetically more enjoyable for students 
who enjoyed ‘the smell of paper’. According to Strli¢ et al. (2009), the aroma 
of an old book is familiar and even appealing to users of a traditional library. 
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With the combination of grassy notes, a tang of acids, and a hint of vanilla over 
an underlying mustiness, this unmistakable smell becomes as much a part of the 
book as its contents (Strlié et al., 2009). 

The distinctive odour of certain books can certainly play an important role in 
aiding the recall of their content (Laska, 2011). Perhaps the popularity of scratch 
and sniff books such as Do Dinosaurs Eat Cookies? by Jane Yolen and Mark Teague 
are testament to the allure of olfactory sensations. With cookie-themed scratch- 
and-sniff lift flaps of scents throughout the book, children can experience the 
aromas of chocolate, cinnamon, strawberry, and more as they turn the pages. The 
multisensory experience adds to the joy, but also to the memory, of the experience. 

Touch too has an important role in reading as hands sense, assist, direct, and 
sustain attention (see Figure 2.1). As hands touch books, the text can be manipu- 
lated, and the pages felt as they are turned — the reader can ‘feel’ where they are 
in the book (Baron, 2015). 

The sense of touch in print adds a spatial ‘thereness’ to text. Readers like to 
know where they are in time and space so they can revisit pages often learning 
from such opportunities (Piper, 2012). This involves the ability to go back to 
check and evaluate the understanding of what was read. 

The sensory stimulation of a print-based book through touch and proprio- 
ception, however, is very different to the haptics of the iPad interface. While an 
iPad can be appealing for young children, is portable, and has intuitive touch- 
screen interfaces, it is also materially hard, unstable, and offers a different tactile 


FIGURE 2.1 Touch too is important for reading 
Photo by Stephen Andrews/Upslash 
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experience. The haptic feedback of a touchscreen is also more challenging as 
children call on fine motor skills to use their fingers for general tapping, preci- 
sion tapping, swiping, and thumb pressing to navigate around the screen — which 
can manifest as a frenzy of haptic engagement for new users (Merchant, 2015). In 
this way, engagement with digital devices demands dexterity and cognitive atten- 
tion to information that might prove challenging for some readers, depending on 
age. So, while there are some similarities between print and screen reading, there 
are also some distinct differences. 

Excess information — or cognitive overload (Kahneman, 1973) — as readers 
tap, swipe, and thumb through a picture book, requires re-allocation of men- 
tal resources available for processing the central narrative towards engagement 
in materials (e.g. pointing, tapping, clicking, and swiping). In a recent meta- 
analysis, engagement with digital devices appeared to attract young children’s 
attention at the expense of a focus on the storyline, even when the content of 
the paper and digital book was the same (Furenes et al., 2021). Increased and 
competing demands on cognitive resources have implications for teaching chil- 
dren to read on digital devices, as well as supporting digital readers when they 
transition to reading to learn. 

The materiality of reading then epitomises the embodied and embedded 
nature of human experience reflecting the multiple entanglements of human 
senses with materials, objects, and artefacts, and the various supports these 
provide to human pursuits (Kallinikos et al., 2012). It may be that print-based 
experiences provide sensorimotor cues that enhance cognitive processing while 
learning to read (Mangen et al., 2019). 


Learning to read 


Learning to read raises some critical issues for decoding texts in digital spaces. 
Toddlers these days spend quite a bit of time looking at screens and may well 
be shown a mobile phone as a pacifier (Chang et al., 2018), later progressing to 
learning to read through digital devices. Familiar children’s stories can be found 
through a range of iPad apps, and bedtime stories can involve a device as easily 
as a printed book (Kucirkova et al., 2013). As noted previously, the materiality 
of experiences and the role of the mind in cognition change when readers shift 
from print to digital reading on screens in a multitude of ways. A good example 
is Jack, a five-year-old ‘beginning’ reader. 

As Jack touches and turns the tangible pages of one of his favourite books, 
gazes longingly at the ice-cream cone in the illustration, follows the linear text 
rhythms with a wandering finger as the story progresses, and then slings the 
colourful artefact under his arm to return it to its rightful place on the book- 
shelf, he actively engages sensory, perceptual, motor, conceptual, and affec- 
tive domains. This much-loved book can be retrieved and read, over and over. 
There is a tactile and engaging experience with the materiality of the book, 
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and the stability of the textual structures. Jack learns through repetition and the 
flow of the narrative, how to predict the storyline, use visual cues, learn com- 
mon sight words, and decode through tangible ‘material anchors’ to eventually 
navigate the text and make meaning (Hutchins, 2005). 

The process of learning to read is difficult. It is not a natural process but 
requires learning to crack the code of written language, decoding meaning in 
texts, and drawing upon a range of cues. Phonics is part of the puzzle; however, 
children also need vocabulary knowledge and text decoding fluency, attending 
to grammar, punctuation, and sentence structure. Digitally based books expand 
access for students but also create different challenges. Tracking the eye move- 
ment of children using digital devices shows that they are more likely to skim or 
read non-linearly, looking for keywords to make meaning, and jumping to the 
end to find out the conclusion; in so doing, the plot and sequence of events can 
be lost. 

For Jack, his favourite printed book is fixed in size and space and anchored 
by the cover. The multiple printed pages include both text and images, pro- 
viding opportunities to engage repeatedly through perceptual, sensual, and 
motor processing, reactivating his memory. When diving into his favourite 
part of the story, Jack can flick through the pages to find his entry point 
quickly. In contrast, digital texts are fluid, often without consistent font size, 
place in space, or anchors to substrates, such as printed items with a cover 
(Rose, 2011). While digital enhancements aligned with the story content 
can support children’s reading outcomes (Christ et al., 2019), digital books 
with unrelated enhancements, such as games embedded in story apps, can 
distract and impact comprehension, taking young children’s attention from 
the task of reading (Munzer et al., 2019). Instead of turning the print-based 
tangible pages, the continuity of reading is disrupted by scrolling the text on 
screen and visual distractions. And while print-based books lend themselves 
to reading over and over, online texts are not necessarily developed for easy 
access and quick retrieval. 

The ritual of engaging in a bedtime story with a parent, or the enjoyment 
of the warmth of a loved one’s lap while exploring a favourite book, is seminal 
to Jack’s experiences (Barzillai et al., 2018; Reich et al., 2016). As such social 
practices unfold, there are often interactions around print text that extend com- 
prehension, such as asking questions, encouraging the use of contextual cues 
to make inferences, and helping Jack to relate texts to his own life (Mol et al., 
2008). These material experiences with books provide ongoing interactions to 
enrich developing reading skills. Interacting with text on digital devices, includ- 
ing e-books and story apps, can be overshadowed by engagement in dynamic 
visuals, exciting hotspots, and endless hyperlinks to games, videos, and creative 
content, altering children’s early literacy experiences. Interpersonal interactions 
are key to digital reading comprehension for new readers (Furenes et al., 2021); 
however, parents and toddlers tend to verbalise less with electronic books. There 
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is less collaboration, and parents read the text less in enhanced electronic books, 
making more format-related comments and negative directives (Munzer et al., 
2019). In this way, new readers’ conversations during digital book reading can 
be dominated by talk about the device or the child’s behaviours, rather than 
the story content (Furenes et al., 2021). That is not to say that children will not 
enjoy reading on screens with a carer or parent — however, the materiality of the 
experience shifts. 


Virtual reality 


Readers can use the external world as an anchor for abstract memories as 
experiences with material objects come to mind and form a background 
tapestry for memory (Schilhab et al., 2018). Visual images of a familiar envi- 
ronment can prompt recall as they involve multimodal experiences of touch, 
odour, light, sound, and so on that can be ignited to recall memory. In this 
way, the mind is involved when children, such as Jack, read favourite books 
and use contextual cues to draw upon prior experiences that arise from 
bodily interactions with the world. For instance, Jack’s previous experiences 
eating an ice-cream cone, involving the senses of seeing, hearing, tasting, and 
smelling along with the fine motor experiences involved in holding, lick- 
ing, and chewing can be simulated in reading engagement through mental 
re-enactment of sensory and motor experiences (Sadoski, 2018). From an 
embodied perspective, meaning making is ultimately grounded in multisen- 
sory experiences. 

It may be that platforms such as virtual reality (VR) that create sound, images, 
video, or other media experiences to give learners a full sensory experience and 
help them engage with the content offer ways to immerse readers by physi- 
cally engaging the senses in a new way. The application of VR technology in 
children’s interactive books has attracted much attention in education due to its 
unique features for providing children with interesting, interactive, immersive, 
and realistic reading experiences (Dong & Si, 2018). VR can already simulate 
sight, sound, and a sense of movement (see Chapter 7), but new technology add- 
ing smell can help stimulate memories and emotional experiences, making VR 
even more real. This works as a device called the ION, which contains vials of 
different scents, attached to a VR headset (see Figure 2.2). As users interact in 
the VR space, scents connected to an experience — say, the act of eating a choco- 
late ice cream — trigger a tiny electric charge releasing the matching chocolate 
fragrance. 

The emerging capability to simulate a wide range of sensory experiences 
beyond sights, sounds, and the touch of conventional digital media, has the 
potential to re-engage future users with the physicality of traditional embodied 
experiences (Ovens & Mills, 2018). The human brain will naturally respond 
to the external stimulus with smell, for instance, closely linked to learning and 
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FIGURE 2.2 Virtual reality encompasses the senses 
Photo by Jessica Lewis / Upslash 


memory. In this way, the seminal role of scent in a scratch-and-sniff book can be 
replicated in the VR space both affording important sensorial experiences but 
also potentially adding further distractions to processing the reading pathway (see 
Chapter 5 for more on olfaction and media). 


Critical issues for digital reading as a mode of making meaning 


Ongoing debates centre on the impact of screens on students’ reading experi- 
ences and their level of engagement with devices. Some studies suggest that 
young people’s high levels of access to mobile phones, computers, and tablet 
devices now means that reading is an activity more likely to be on screen than on 
the printed page — escalated by COVID-19 lockdowns and reliance on remote 
learning (National Literacy Trust, 2020). Others hold fast to the idea that read- 
ers prefer print, especially in academic contexts (Baron et al., 2017). And while 
middle school students, particularly middle school boys, favour recreational 
reading in digital settings (McKenna et al., 2012), others insist that younger 
children prefer reading print-based books, particularly in their leisure time 
(Kucirkova & Littleton, 2016). Adding complexity, reports of reading compre- 
hension on devices have had varied results. Some students read faster on devices 
and believe they comprehend efficiently through digital mediums though their 
assessed understanding of texts is more sophisticated when they read from print 
(see Clinton, 2019; Kong, Seo & Zhai, 2018; Singer & Alexander, 2017). Others 
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report differences in reading comprehension depending on the task and the qual- 
ity of the interaction, and point to technology advancements (e.g. artificial intel- 
ligence, augmented reality, virtual reality) as ways to improve meaning making 
(Furenes, Kucirkova & Bus, 2021). 

The negative effects of digital devices on reading are not agreed upon, and 
some researchers believe that biological plasticity and bodily reconfiguration or 
re-embodiment will occur as individuals adjust to the affordances of new devices 
(Schilhab et al., 2018). One moderator of digital text comprehension could be 
reader experience using technology — and that comprehension difficulties will 
disappear once students are savvy with technologies (Delgado et al., 2018), or 
that their brains will reconfigure for processing on different devices (Schilhab 
et al., 2018). 

In the meantime, digital text types are something that needs to be considered 
by educators. It may well be that when the texts are short — a page or less — and 
comprehension demands are light (grasping the main idea), students do well with 
both informational and narrative digital texts. But as learning demands increase 
and the texts are more extensive, paper-based reading affords better understand- 
ing. The level of skill required depends on a number of variables related to the 
task (Coiro, 2011). The superiority of print shines when recall requires going 
beyond questions that have superficial answers to those that require inferences, 
details about the text, or remembering when and where in a story an event took 
place (Baron et al., 2017). In understanding reading behaviours in digital spaces 
there are several factors that are important to consider, including whether the 
reading is done with time pressure, the length of the text, as well as the depth of 
processing that the task requires (Delgado et al., 2018). These aspects of texts then 
interact with the characteristics of individual readers to shape reading behaviours. 


Reading behaviours that impact on meaning making 


With an increasing amount of time spent reading digitally, screen-based read- 
ing behaviours are emerging — with tendencies for browsing, scanning, key- 
word spotting, one-time reading, and non-linear selective reading, with less 
in-depth or concentrated reading (Liu, 2012). Multimodal semiotic awareness is 
also important for making meaning of digital texts. Semiotic resources, such as 
layout, composition, text, image, graphics, and the range of colours and fonts, 
can be distracting. Multimedia elements, animations, and interactive features can 
enhance the reading experience, but also require filtering and making choices 
about reading pathways. Nicholas et al. (2008) refer to such reading behaviours 
in the context of searching digital libraries as ‘power browsing’, where readers 
seldom sustain engagement with text online for a long period of time. These 
strategies do not involve going deeper to understand the complexity of issues. 
In a print-based environment, the author determines the order or presen- 
tation of ideas — in hypertext environments the author provides options, but 
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readers choose their personal pathway by activating hyperlinks. This non-linear 
reading (e.g. jumping from page to page and from site to site), can reduce the 
sustained attention to any one textual source and lead to more fragmented read- 
ing. While traditional print-based texts afford many stable material anchors to aid 
memorising and comprehension, digital texts are more cognitively demanding. 
Continual interruptions occur with disruptions from popups, alerts, notifica- 
tions, hotspots, or links. Reading behaviours increasingly include ‘stacking’ — using 
multiple devices for conducting unrelated tasks — and ‘meshing’ — simultaneously 
communicating content being viewed (Davidson & Harris, 2019). This lack of 
engagement with digital material is illustrated by young people’s use of the labels 
TL;DR (too long; didn’t read) and SR;MP (skim-read; missed point). These 
behaviours and approaches to reading on screens can have negative effects that 
appear as early as fourth and fifth grades (Golan et al., 2018), and need to be 
considered in contemporary curriculum and pedagogy. 

A paradigm shift is needed to support reading processes in digital spaces. 
Digital information presents both new richness and novel challenges for the 
mind — as the fluid, multimodal nature of the digital allows immersion in dif- 
ferent modalities. Shifts from stable, linear texts to non-stable modalities require 
shifts in attention, decision making, and complex cognition to navigate personal 
reading pathways through hyperlinked spaces. Such cognitive functions relate to 
self-regulation and processes that enable us to plan, focus attention, remember 
instructions, and juggle multiple tasks successfully as the brain filters distractions, 
prioritises tasks, sets and achieves goals, and controls impulses (Kieffer et al., 2013). 
Children are not born with these skills — but they are born with the potential to 
develop them. It is these cognitive functions that are key to supporting reading 
and making meaning in digital spaces. 


Implications for literacy curriculum and pedagogy 


Digital reading is now an important part of daily life for many students in class- 
rooms across the globe. Due to the complexities of engaging with such texts, 
digital reading needs to be explicitly addressed with strategies systematically 
taught through the primary years as children learn to read, and then in middle 
and secondary years, as children read to learn. Although we do not fully under- 
stand the source of differences between digital reading and print-based reading, 
there are fundamental changes in experiences (Afflerbach & Cho, 2010; Leu & 
Maykel, 2016). Two significant moderators are important to consider. First, the 
time frame allocated for a task is critical as the advantage for paper-based reading 
is stronger in time-constrained reading than in self-paced reading; and the text 
genre makes a difference as the paper-based reading advantage is consistent 
across studies using informational text or a mix of informational and narrative 
texts, but there is often no difference for narrative-only texts (Delgado, Vargas, 
Ackerman & Salmerón, 2018). 
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There are a range of strategies, however, to address concerns related to how 
digital reading differs from print-based reading — particularly issues of focusing 
attention, deep reading, and how adults can scaffold experiences. Research is 
a long way from any definitive answers; however, educators can look towards 
emerging approaches to support their students’ digital reading. 


Scaffolding digital reading in the early years 


Many well-meaning schools have embraced technologies and incorporated the 
latest array of tools and apps into children’s learning, despite a lack of research 
on their efficacy or guidance on how best to support their use (Kucirkova & 
Littleton, 2016). In an increasingly digital environment, young readers need to 
develop specific screen-based reading behaviours (e.g. skimming, browsing, key- 
word spotting, bouncing between multimodal texts), to cope with large amounts 
of texts and the abundance of information, and to overcome surface level strate- 
gies. The role of educators is critical for supporting new readers in digital spaces. 
Shared reading using digital texts does not offer the same rich language and 
bonding experience that occurs when children have close contact with a caring 
adult who scaffolds the reading journey. To address these differences in adult 
interactions when using these mediums, and the lack of dialogic reading that 
naturally unfolds through sharing print-based texts, educators need to explicitly 
scaffold and support children’s decoding of texts for meaning making (Strouse 
et al., 2019). 

The screen-inferiority effect on reading outcomes is strongest in the school 
context where group-based activities make it hard for children to engage with 
digital enhancement to support their decoding (Hoel & Tonnessen, 2019). 
‘Read to me’ built-in functions offered by e-books and apps can be a problem 
when teachers utilise these modes for children’s independent use of tablets and 
iPads, rather than engaging in joint reading experiences (Kucirkova & Littleton, 
2016). For example, de Jong and Bus (2002) found that kindergarteners allowed 
to independently use electronic storybooks barely listened to the audio narrative, 
and navigated the story in a suboptimal manner, spending nearly half their time 
playing games. Therefore, children need adult interactions, perhaps even more 
in digital spaces, as they are confronted with so much choice and distracting 
features. 

Building on print-based interactions between a child and adult can enliven 
digital spaces — with a focus on questioning, making links, and scaffolding the 
reading experience. In this way, reading is reconceptualised as an extension of 
print-based experiences. Adults can engage in discussions around the meaning of 
words to support children’s efforts to relate the story to their own experiences, 
and to ask questions to identify events, characters from the story, and sequences 
of events, advancing superior recall while extending their reading communities. 
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With digital multimedia books, such as the ME Books app, personalised responses 
to a story can be embedded into the book as children or adults audiorecord their 
own story narration to become part of the story. In this way, digital books can 
outperform paper books when they include enhancements that increase chil- 
dren’s meaning making of the narrative, for instance, by eliciting children’s back- 
ground knowledge or providing additional explanations of story events (Furenes 
et al., 2021). The use of open-ended apps can lead to more collaboration among 
children and higher-order talk including the use of questions, listening and 
building on each other’s ideas. 

New aftordances are also provided through remote connection for children, 
families, and teachers for daily reading and online platforms for interactive read- 
ing communities. Online platforms, such as Twitter, Facebook, Instagram, 
Bookstagram, email, and school websites, along with communities, such as 
Biblionasium, set up spaces for young readers with virtual bookshelves that allow 
children to add what they are reading, want to read, and have already read. 
Sharing favourite reading preferences with peers, teachers, librarians, and family 
who all bring different ideas and experiences can be enriching, enjoyable, and 
extend reading experiences. Communities have never been more important as 
they can support readers, provide an online space to share ideas, enrich read- 
ing experiences, and through recommendations, discussion, and debate, become 
part of the digital reading experiences. 


Supporting deep reading to learn in the later years 


As students transition and read to learn across devices, reading in spaces such 
as multimodal websites changes the experience of reading (Afflerbach & Cho, 
2010; Leu & Maykel, 2016). The explicit instruction of comprehension strategies 
(predicting, thinking aloud, text structuring, visually representing text, summa- 
rising, and questioning) in a scaffolded digital environment, can prepare students 
with digital literacy skills to competently display their knowledge on computer- 
based measures of reading comprehension (Ortlieb et al., 2014). Students can 
also learn strategies to enhance decoding texts to make meaning and slow down, 
attend carefully to key criteria, and know how to filter information (Singer & 
Alexander, 2017). Teachers can find ways to capture and understand the full 
range of students’ abilities so they can guide learners toward more strategic use of 
practices that will better prepare them to make sense of information in different 
online spaces (Castek & Coiro, 2015). 

Deep reading involves slow, immersive processes, whereby readers take time 
and cognitive effort to engage in deep thought (Wolf et al., 2012). Digital 
spaces do not naturally encourage taking time (as readers jump around hyper- 
linked sites) or investing cognitive effort, as readers skip from one point to 
another without evaluation. Deep reading cognitive processes associated with 
print reading, however, can be translated to digital reading practices with the 
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support of educators (Lauterman & Ackerman, 2014). To overcome the pro- 
cessing of information in short bursts, educators can help develop reading 
practices and encourage students to: 


e Carefully plan the reading pathway; 

e Choose credible digital sources; 

e Focus attention related to the task; 

e  Prioritise tasks related to the goal; 

e Re-orient after unplanned detours and distractions; 

e Revisit instructions, map progress, and monitor pathways; 

e Filter distractions due to hyperlinks, popups, media, and advertisements; 
e Think flexibly and evaluate the usefulness of information. 


Annotating and mind-mapping are important tools and note-taking can help 
summarise main ideas. This can be as simple as using Adobe Acrobat for stu- 
dents to highlight main points, apply sticky notes, and make comments on 
documents. Or it may mean creating a print-based web of a personal reading 
pathway across multiple digital sites. These skills are considered more closely in 
Chapter 4 with attention to how skilled readers need to simultaneously draw 
on print-based reading processes with more complex applications. Explicit 
strategies are required in the transformation to support focused, sequential, 
text-centred engagement instead of fragmented, restless, grazing behaviours of 
clicking and scrolling (Wolf & Barzillai, 2009). This type of reading, however, 
requires students to focus and sustain attention on a task. Reading online across 
multimodal websites can also be scaffolded by adult interactions as personal 
reading pathways are planned, revisited, and monitored. Social experiences 
and interactions that extend to interaction with other online experiences (e.g. 
gaming communities, online communities, or social connections) can bring 
about connections between prior experiences, interests, and communications 
with others online. 

Educators cannot assume that students are competent because they have 
grown up immersed in a world dependent on technology, and they cannot 
anticipate that print-based reading skills automatically translate to digital spaces 
— such skills need to be explicitly taught so that as students progress through 
the school years they can decode texts, make meaning, and engage in learning 
across a range of digital devices. There is some urgency in addressing reading 
behaviours and advancing student digital reading skills as high-stakes assess- 
ments move to digital formats. International assessments, such as the Programme 
for International Student Assessment (PISA) (OECD, 2015) and the program 
known as Progress in International Reading Literacy Study (PIRLS) (Mullis 
et al., 2017) are instituting digital administration as well as scenario-based tasks 
that incorporate digital literacy, with consequences for readers. This approach 
is also echoed across various nations. For instance, to address the increased role 
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of technology in classrooms, the National Center for Education Statistics in the 
US, is transitioning the National Assessment of Educational Progress (NAEP) 
from paper and pencil to digital assessments. This includes assessment of reading 
which is administered on tablets or laptop computers, raising issues of decoding 
and comprehension through such modes. 


Recommendations for research of digital reading practices 


Digital texts are here to stay for all kinds of economic, social, cultural, and edu- 
cational reasons. This chapter has pointed to how readers increasingly face new 
challenges in making meaning of texts in digital spaces that change the material- 
ity and embodiment experience. 

Research of reading practices in coming years will require transformed under- 
standings of how reading practices metamorphose in digital spaces. To date, the 
majority of studies fail to clearly define or conceptualise what constitutes digi- 
tal reading (Singer & Alexander, 2017). As yet, there is no cohesive theoreti- 
cal framework that concurrently describes the different cognitive, motivational, 
and metacognitive processes involved in reading across different digital devices 
(Golan et al., 2018). There is some urgency in specifically defining digital read- 
ing, and recalibrating what it means to read in digital spaces. This means consid- 
ering the issues raised in this chapter to expand traditional conceptions of reading 
for meaning making in ways that appreciate the complexities of digital texts and 
that situate literate practices on a continuum (Moje, 2009). 

In light of the ongoing evolution in technologies, further research related to 
differences in decoding texts across multiple devices is needed. Current research 
in the early elementary years of schooling has focused on narratives (Singer & 
Alexander, 2017) with little known about student experiences reading informa- 
tional text, which require higher-level processing, such as using complex aca- 
demic vocabulary and structures, with less connection to real life. ePIRLS, an 
extension of PIRLS (Mullis et al., 2017), uses computer-based assessment focus- 
ing on informational reading to assess fourth-grade students’ ability to use the 
internet in a school context. This focus is important for student skill sets in the 
future. What kind of reading practices ensure success in decoding conflicting 
sources of information on the web? Are these skills different to literacy practices 
involved in decoding to make meaning to solve problems, and to succeed in 
video games? These two very different kinds of textual decoding will be consid- 
ered in Chapters 3 and 4. 
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CRITICALLY EVALUATING MULTIPLE 
SOURCES FOR DIGITAL FUTURES 


Why critical reading of online sources is important for 
digital futures 


In the rapidly changing digital world, the internet is host to almost endless 
amounts of controversial information that young people can access at school 
and in their leisure time. This wealth of diverse and accessible information offers 
exciting opportunities to be immersed in information from around the globe 
with multiple perspectives and viewpoints on a range of socially relevant topics 
available at any time. However, conflicting sources abound, particularly when it 
comes to information related to socio-scientific issues that are by nature social 
issues informed by science (Zeidler et al., 2019). For young people to engage 
with evaluative thinking, and to make informed decisions about such socially 
relevant, real-world problems, they need a sophisticated epistemic stance (know- 
ing about knowledge) to apply as they read, synthesise, make inferences, and 
justify their interpretation of multiple online sources. We argue this processing 
is a new way to think about critical literacy and refer to this approach as critical 
epistemic literacy. 

Personal approaches to knowledge are important and affect both processes and 
outcomes of student learning through online sources (Braten et al., 2011; Hofer & 
Pintrich, 2002). Futures-focused critical literacy research should pay attention to 
the important role beliefs play in students’ meaning making about controversial, 
socially relevant real-world problems as they make decisions about the trustwor- 
thiness of websites and the integration of information to construct new knowl- 
edge (Barzilai & Zohar, 2012). While not all scholars agree on approaches to 
understanding epistemic beliefs, there is a shared view that personal beliefs about 
knowledge and knowing play an important role in formal and informal learning 
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(Hofer & Pintrich, 2002), and that many students would benefit from a more 
sophisticated stance (Mason et al., 2011). This is particularly true for students 
engaging with information related to socio-scientific issues. 

More than ever, young people around the world are engaged in social issues — 
including climate change, health pandemics, and phenomena related to genetic 
modification — which requires new epistemic forms of critical literacy. What is 
emerging is a range of scholarly work examining the cognitive strategies and 
skills adolescent readers use as they navigate these digital spaces in search of 
information (Castek & Coiro, 2015). As well as these cognitive decoding skills, 
students also require sophisticated personal epistemologies and processes to con- 
struct knowledge (Cho et al., 2018). As our world becomes more networked, 
connected, and smaller in the digital age, new approaches to critical literacy are 
essential to keep pace with evolving technologies and the innovations that are 
creating complexities in processing and making meaning in digital spaces. 

Fundamentally different to accessing knowledge through traditional print- 
based books, multimodal texts related to socio-scientific issues on the web offer 
multiple accounts that differ in scope, argument, and evidence, with sources 
also varying in their purpose, credibility, and authorship (see Chapter 8 on 
Infographics and Scientific Literacy). New approaches to literacy can build on 
traditional reading skills to include critical thinking informed by sophisticated 
epistemology. In this way, approaches to reading to learn about topical issues online 
require some recalibration in terms of what it means to make meaning as these 
texts are not simply substitutes for print-based materials (Furenes et al., 2021). 

Such shifts call for rethinking of what constitutes critical literacy for engag- 
ing with socio-scientific issues on the internet. This chapter will explore the 
intersections of evaluative reading, epistemic thinking, and critical digital litera- 
cies. Understanding how beliefs shape evaluative judgements is rarely discussed 
in critical literacy approaches yet is the underlying foundation for developing 
advanced critical readers. We need to consider what thinking skills should be 
foregrounded as essential for literacy in the classroom and in societies of the 
future. To this end, this chapter explores the importance of young people learn- 
ing the skills to successfully think, evaluate, and make judgements about con- 
flicting sources of information related to topical social issues as they engage in 
inquiry learning across the curriculum and develop into active citizens in the 
digital world. This chapter highlights some of the recent developments in under- 
standing the role of epistemic approaches to knowledge and exciting opportuni- 
ties for advancing young people’s stance as a way of recalibrating critical literacy. 


Socio-scientific issues in the networked world 


Never before have young people been so involved in socially relevant problems 
informed by science. Engagement with socio-scientific issues is now part of 
daily life for many children and adolescents across the globe. The rise of Greta 
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Thunberg as a ‘famous’ climate change activist has perhaps been pivotal in insti- 
gating young people from countries around the world to take a stance on such 
issues, walk out of class, and march in the streets to demand action. Where pre- 
viously young people talked about climate change, the latest generation is pro- 
testing loudly with strong convictions about their interpretations of the science 
that they make visible on social media and various web platforms. In the context 
of issues such as climate change, however, contradictory information, misin- 
formation (false), disinformation (misunderstood), and advocacy for competing 
theories abound. Much of this ‘information’ is translated on the net through 
international and government agencies, activist organisations, and blogs, but is 
particularly rampant via social media (Lazer et al., 2018). As information is not 
knowledge, students often have limited literacy skills to make the evidence-based 
judgements needed to engage as informed and active citizens in such contexts. 

As countries look towards schools as a vehicle for teaching about climate 
change (Beach et al., 2017), nations such as Italy have made sustainability and cli- 
mate crisis a compulsory subject for school children, and New Zealand’s schools 
are looking to teach students about climate crisis, activism, and ‘eco anxiety’ 
(Erskine, 2019). These approaches bring to the fore increasing impetus for teach- 
ing critical epistemic literacy. It is essential that young people have a “cognitive 
repertoire’ — to understand the nature of disinformation to make informed deci- 
sions when there is high level of disagreement among experts (Van der Linden 
et al., 2017) — as well as relevant disciplinary knowledge and critical processing 
skills for understanding the complexity of issues such as climate change and how 
to weigh up the reliability of evidence. 

An online context where conflict and controversy over information is evi- 
dent has been heightened by the global pandemic. The internet provides almost 
unlimited links to opposing arguments regarding COVID-19, from disagreement 
over the effectiveness of face masks in preventing viral transmission, to compet- 
ing claims about the promise of certain vaccines and treatments. A recent Google 
search of the term “corona” returned 3,334,000,000 results (see Figure 3.1) provid- 
ing almost unlimited potential for misunderstandings. Confusion and the rapid 
flow of varying information impacts young people at school, in their home, and 
each and every day in their community, polarising people with different points of 
view. Instigating ongoing protests and marches against approaches to managing 
COVID-19, young people’s beliefs should be evaluated based on the evidence 
presented to make informed decisions. This involves the skills to read, decode, 
synthesise, and evaluate conflicting sources of information that evolve daily. 

The rise of ‘fake news’ highlights the erosion of gauges to filter the trustwor- 
thiness of information in the internet age — with concerns about the spread of 
disinformation now global (Lazer et al., 2018). Much remains unknown regard- 
ing the vulnerabilities of young people engaging with new forms of informa- 
tion regulated by algorithms or artificial intelligence that produce biased sources, 
with new approaches to savvy information consumerism needed. As young 
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FIGURE 3.1 Google search results for the term ‘corona’, November 2021. The high 
number of search results is shown, with 3,340,000,000 matches 

Note: Google is a trademark of Google LLC and this book is not endorsed by or affiliated with 
Google in any way 


people engage with online information around socially relevant real-world prob- 
lems, without sophisticated processing strategies for directing personal pathways 
and managing the trustworthiness of content, they are bombarded with more 
of the same. This is because ‘algorithmic bias’ — a set of rules modulated by 
computer programs to determine the order in which search results are presented — 
promotes online content that is aligned with consumers’ previous views rather 
than with trustworthiness, amplifying the psychological phenomenon whereby 
people prefer to consume information that matches their belief systems. This is 
known as ‘confirmation bias’ (Bessi et al., 2014). Internet platforms are also the 
most significant enablers and disseminators of ‘fake news’ related to topical social 
issues (Lazer et al., 2018). Reading to decode and learn from sources on the 
internet then presents some challenges, particularly when dealing with conflict- 
ing sources related to controversial issues. 


Investigating socio-scientific issues 


Let’s take the example of Josie, an adolescent who has been asked to investigate 
health concerns related to mobile phone use by her middle school science teacher. 
This socio-scientific issue now confronts adolescents in the ‘knowledge/risk’ soci- 
ety where scientific knowledge plays an increasingly significant role in their lives. 
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The controversy here centres on potential health risks and competing claims 
related to radiation, the ‘thermal’ effect, and possible links to cancer. As such, 
mobile phone research is ‘science in the making’ (Latour, 1987), where young 
people may have to reason and justify their standpoints concerning trustworthi- 
ness and decision making on issues connected to health that are ever-evolving. 
Josie, then, during her inquiry task will be faced with conflicting evidence, theo- 
ries, and views about the topic. While some sources claim that low levels of 
radio-frequency radiation emitted by mobile phones cause health problems such 
as headaches or brain tumours, other sources will indicate that no conclusive evi- 
dence exists that mobile phones are damaging to health in the short or long term. 

As Josie approaches the inquiry task by googling mobile phone health risks 
on the internet, ideally, she draws upon a sophisticated understanding of knowl- 
edge to decode texts, navigate hyperlinks, and construct personal reading paths 
to access, synthesise, and reason about her findings in a multimodal digital space 
that can make critical evaluation challenging. Reliance on Google is increasing 
and as of June 2021, this search engine had a market share of 88% (Johnson, 
2022) making this mode of exploration an everyday event for many young peo- 
ple. Such an approach in a hyperlinked space brings with it a lot to navigate. 
As well as print-based skills such as previewing the linear text, making predic- 
tions while reading, interpreting the meaning, making connections within and 
between texts and integrating textual clues with background knowledge, Josie 
also has to self-regulate her reading pathway amidst endless amounts of infor- 
mation available on the web, avoid distractions by hyperlinks, videos, pop-up 
advertisements, and the tendency to skim and scroll through texts and evaluate 
the sources. There is a lot to coordinate and navigate. 

Biased information that appears after a search on Google is increasing as the 
platform interferes with search algorithms and changes results. Google is cur- 
rently a leader in artificial intelligence (AI), raising problems of bias through 
its algorithms. Along with promoted sites, algorithms can also adopt personal, 
political, and information biases (Epstein & Robertson, 2015). Not surprisingly, 
companies now spend billions of dollars annually in efforts to be placed at the top 
of search result lists (Econsultancy, 2012) as people generally scan search engine 
results in the order in which they appear and then become fixed on the results 
that rank highest, even when lower-ranked results are more relevant to their 
search (Granka et al., 2004). 

Once Josie selects her first web page, she must then decide which hyperlinks 
she wants to navigate, and in which order. Along with promoted sites, Josie may 
well continue to encounter the same views that she found in her initial search. 
Efficient navigation can be influenced by sequencing pages that are relevant 
for the goal to maintain high semantic coherence between the current and the 
linked page, avoiding big ‘semantic jumps’, however, Josie may also decide to just 
follow her interest or scan aimlessly (Salmerón et al., 2020). Once she has been 
confronted by several mixed modality sources representing different perspectives 
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or contradicting information about mobile phone health risks, Josie’s goal will 
be the often-challenging task of integrating information across multiple sources 
(Salmerón et al., 2020). In this way, Josie will become the author of an integrated 
mental representation by selecting and integrating different pieces of information 
(Afflerbach & Cho, 2009). 

Josie’s ability to distinguish the relative trustworthiness of the knowledge 
claims may rely on her quest for one right answer, or her first-hand knowl- 
edge as Josie compares what she has read about mobile phones against what she 
already believes to be true. People tend to prefer information that confirms their 
pre-existing attitudes (selective exposure), as this information is more persuasive 
than dissonant information — they are much more inclined to accept desirable 
information that aligns with their existing point of view (Lazer et al., 2018). 
Alternatively, Josie may come to a decision by scrutinising the source of informa- 
tion and questioning who to believe. If she does rely on source information to 
make trustworthiness judgements, she may consider the expertise or intentions of 
a source, the date of publication, and the extent to which information accuracy 
is assured through some criteria for quality check, however, she is more likely to 
rely on rather superficial cues, such as a professional-looking design (Salmerón 
et al., 2020; Stromso & Braten, 2014). In the end, how Josie approaches her 
mobile phone inquiry task will be informed by her beliefs about knowledge. 
Such beliefs will, in turn, influence her learning (Hofer & Pintrich, 2002). 


New directions for critical literacy online: the epistemic turn 


As students with sophisticated approaches to knowledge can significantly out- 
perform students with absolutist or black and white thinking (Barzilai & Zohar, 
2012) developing epistemic approaches offers a new direction to inform criti- 
cal literacy for online integration and construction of knowledge. To develop 
these sophisticated processing skills, we first need to identify the demands on 
individuals accessing conflicting sources of information online, the processes that 
support such inquiry, and the role of personal epistemology in the evaluation of 
both website trustworthiness and the integration of multiple accounts. Then we 
should turn our attention to how critical epistemic literacy can be advanced for 
online readers. 

Advanced readers actively construct meaning through a set of powerful reading 
comprehension strategies when confronted with print-based texts, however, we 
are now learning about the new skills readers use to decode and construct knowl- 
edge on the internet (Barzilai & Zohar, 2012; Cho et al., 2018). While there are 
similarities in the processes required to comprehend via print and on devices, 
decoding to make meaning in digital spaces is also more complex (Coiro & 
Dobler, 2007) and involves evaluative reading practices (Fang & Schleppegrell, 
2010) along with epistemic skills for self-regulation and processing (Strømsø & 
Braten, 2014). 
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Skilled internet readers draw on their knowledge of the topic and printed 
informational structures to guide their reading pathways, using inferential rea- 
soning strategies informed by their use of literal matching skills, structural cues, 
and context cues as they choose what they read, drawing on traditional self-regulated 
reading processes such as goal setting, predicting, monitoring, and evaluating 
(Coiro & Dobler, 2007). They also use new, more complex dimensions of read- 
ing comprehension as they navigate hyperlinked digital spaces that offer complex 
website structures and require forward inferential reasoning across multiple and 
multimodal texts, with the cognitive flexibility for rapid self-regulated reading 
processes (Mason et al., 2011). 

Hyperlinked internet spaces change how information is accessed, organised, 
and presented. Importantly, young people also need knowledge of sourcing cred- 
ible and reliable information on the web (Strømsø & Bråten, 2014). Hypertext 
environments require readers to make choices from a number of alternative 
sources initially identified by a title, an excerpt of text from a web page, authors’ 
credentials, the logo of the institution that hosts the website, and URLs. While 
the wonder of the internet is the almost endless amount of information available 
at one’s fingertips, sourcing the ‘right’ information requires navigation, while 
evaluating it requires accessing, synthesising, and reasoning about often con- 
flicting views. Reading to learn and construct knowledge increasingly involves 
evaluative reading practices that rely on epistemic cognition for successful pro- 
cessing (Cho et al., 2018). 


Epistemic cognition 


To appreciate the role of epistemic cognition in critical approaches to literacy it 
is important to understand theoretical approaches to epistemology. Epistemology, 
from the Greek, ‘study of knowledge’, has focused on defining what constitutes 
knowledge and discerning it from opinion, faith, and misinformation, while epis- 
temic cognition is a body of work that refers to how people think about what they 
know (Sandoval et al., 2016). This approach to cognition is gaining increasing 
recognition in relation to young children’s reasoning (Lunn et al., 2017; Scholes 
et al., 2021), comprehension of multiple texts (Braten et al., 2011), and argumen- 
tation skills (Mason & Scirica, 2006), with some promising implications for online 
reading and inquiry (Barzilai & Zohar, 2012). In a ‘post truth’ era of alternative 
facts, enhancing cognition in the digital world would seem a priority. 

Epistemic cognition is activated at multiple points during online inquiry and 
in judgements made (Mason et al., 2011). Judgements related to sourcing and 
processing information may be informed by simplistic understandings of knowl- 
edge or more sophisticated approaches to evaluation. Drawing on Kuhn and 
Weinstock (2002), epistemic cognition involves personal beliefs or perspectives 
that can be described through a range beginning with objectivist beliefs, moving 
towards subjectivist beliefs, and then finally developing into sophisticated evaluativist 
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beliefs (see Table 3.1). These beliefs then become ingrained in a reader’s stance 
as they approach and process multiple sources online. For instance, in terms 
of Josie’s task related to mobile phone health earlier, her approach to sourcing 
information would be influenced by her specific stance. 

If Josie was to approach reading conflicting sources online from an objectivist 
stance, she may seek out one source of truth that is certain. Online searching may 
be simplistic and brief as she may not see the value in searching for additional 
websites in order to integrate information or to deliberate about the credibility 
of the online sources (Kuhn & Weinstock, 2002). There would be belief in an 
objective reality which is directly knowable and a certainty of knowledge so that 
Josie looks for ‘real’ science without judging credibility or competing claims. 
A reader who views knowledge this way seeks a single source for information 
or a range of confirmatory information about a complex topic and accepts this 
as fact. This then articulates into ‘confirmation bias’ (Bessi et al., 2014). Readers 
justify their source of information based on the authority of the site for convey- 
ing the right answer or the ‘truth’ (Barzilai & Zohar, 2012). 

As epistemologies advance, Josie may approach reading conflicting sources 
online from a subjectivist stance, approaching knowledge with a belief in many 
right answers. This inclusive approach values personal opinions and different per- 
spectives as knowledge within individuals, and therefore as subjective and pro- 
tected from judgement (Kuhn & Weinstock, 2002). Josie would believe that there 
are many right and valid answers to a problem without any evaluation of these 
diverse points of view or weighing up of the evidence. This approach can lead to 


TABLE 3.1 Epistemic approaches to knowledge 


Epistemic stance Approach to knowledge 


Objectivist stance e One right answer, black and white thinking about knowledge 
e Apply simplistic strategies to source the ‘truth’ 
e Limited consideration of quality of sources with justification 
based on passive reception from authority 
Subjectivist stance e Many right answers as knowledge within individuals so all 
perspectives valid 
e Apply fragmented strategies to source ‘multiple’ truths 
e Consider a range of different sources with justifications based 
on personal criteria 
Evaluativist stance ¢ Evaluate and construct knowledge 
e Apply criteria to navigate sources and personal pathway 
e Need to compare and contrast across sources to make 
judgement to integrate multiple sources to construct 
knowledge 


Note: The table lists the key differences in epistemic approaches to knowledge. The left-hand column 
lists objectivist, subjectivist, and evaluativist stances. The right-hand column lists three features of 
each of these approaches. 
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untethered, fragmented searching on the internet and restless grazing behaviours, 
with no defined pathway or criteria for searching as all opinions are deemed 
important. She may rely on simple criteria for inclusion such as website design, 
author credentials, domain names, or publication date with justification for her 
sources based on many right answers and personal opinions (Salmerón et al., 2020). 

Once sophisticated epistemic beliefs are developed, if Josie takes an evaluativ- 
ist stance — she would justify online knowledge by comparing multiple sources, 
drawing on criteria for evaluating conflicting perspectives and considering the 
reliability of the information she draws upon (Mason et al., 2011). The shift to 
an evaluativist stance includes the need to coordinate objectivist and subjectivist 
dimensions of knowledge as knowledge is constructed, can be uncertain, and 
is always evolving (Kuhn & Weinstock, 2002). Evaluativist thinking is a belief 
in the idea that knowledge is changeable and that ideas can be evaluated from a 
range of perspectives with the potential for new ideas to be constructed. 

From this evaluativist point of view, there is an integration of sources of infor- 
mation and engagement in evidence-based reasoning to inform decision making. 
Approaching online inquiry from this stance may result in taking a critical gaze 
to sources and adopting thinking that involves evaluation of conflicting evi- 
dence based on criteria to establish trustworthiness and more awareness of bias 
(Barzilai & Zohar, 2012). Josie’s justification of knowledge (e.g. why she believes 
a website to be credible) is associated with her perspectives on what counts as 
scientific knowledge and how to approach science learning — rather than finding 
the right answer, she conducts evaluative sourcing and integration of different 
perspectives that result in more advanced understanding or knowledge construc- 
tion (Mason et al., 2011). 

Ranging views, or epistemic stances, come into play during everyday actions 
that involve knowledge judgements and knowledge constructions, potentially 
informing how young people approach online reading and inquiry (Barzilai & 
Zohar, 2012). This becomes a manifestation of “theory-in-action” (Kuhn & 
Weinstock, 2002, p. 134) as readers approach intertextual online spaces. Locating 
relevant information effectively and efficiently is undoubtedly a crucial ability, 
but it must be accompanied by an ability to assess the value of online materials to 
come to reasoned conclusions. When applied to understanding socio-scientific 
issues that present opposing views and offer conflicting explanations, deciding 
what to believe and what counts as knowledge or who can be trusted as a source 
of information requires critical application and thinking (Greene et al., 2016), 
which are skills that can be taught in early elementary school (Lunn et al., 2017: 
Schiefer et al., 2020). 


Beliefs in action 


Epistemic processing is particularly significant for inquiry related to socio- 
scientific issues, due to the uncertainties and possibilities of the non-linear, 
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intertextual nature of the internet (Mason et al., 2011). Such approaches online 
include personal perspectives translated into actions of individuals (Kuhn & 
Weinstock, 2002). Cho et al. (2018) refer to the way beliefs about knowledge 
and knowing are activated during online reading; accordingly, more success- 
ful online readers tend to engage in higher-order thinking processes when judg- 
ing information sources, monitoring their knowing processes, and regulating their 
alternative knowledge-seeking actions. Cho et al. (2018) describe the way these 
three processes need to be advanced to improve online reading. 


e Making judgements. This process concerns readers’ ability to evaluate the nature 
and quality of a source such as trustworthiness. It also includes awareness of 
how sources interact with one another (Do they agree or conflict?) and per- 
sonal viewpoints (Do sources make sense or challenge my beliefs?). Successful 
readers notice source authorities and integrate multiple viewpoints to judge 
the legitimacy of certain ideas and claims and the sources that represented 
them, while using evaluative thinking to help them access useful information, 
assess source reliability, and construct understandings from sources. 

e Monitoring searches. This process concerns readers’ approach to monitoring 
the decisions they make when searching. It includes choosing where to 
start first, where to search next, and personal pathways — critical reflection 
on progress and views on the information intentions. Successful readers 
approach knowledge as complex and contextual rather than simple and cer- 
tain, and they think of themselves as active agents able to interact with the 
situation of reading and to learn from available texts by using the knowledge. 

e Regulating knowledge. This process concerns readers’ regulation of the process 
of controlling possible actions that can be taken to support confirmation of 
knowledge and seek out more of the same or additional perhaps conflicting 
knowledge — evaluation of alternative approaches. Successful readers think 
about their actions to address their lack of knowledge, revise their current 
understanding, and build on what they know by accessing additional sources 
of information. 


The key point here is that learners with sophisticated beliefs are more flexible 
in their adaption of learning strategies for making judgements than readers with 
naive beliefs (Bromme et al., 2008). They may have more strategies to draw 
upon and demonstrate more complex integration of information to construct 
knowledge — important factors in processes and outcomes of learning (Braten 
et al., 2011). From this perspective, advancing students’ epistemic stance in the 
elementary school years would be a priority in educational contexts. 


Implications for educational practice 


Epistemic engagement is undeniable in the context of the rapid circulation of 
multimodal texts from varied knowledge sources, however these higher-order 
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skills — interpreting, making inferences and integrating multiple documents — 
particularly in relation to pressing socio-scientific issues, can be explicitly taught 
in the classroom as a new dimension of critical literacy. The internet potentially 
fosters student thinking as they access intertextual sources in a non-linear space 
to choose what to read, when to read, and navigate their personal pathways to 
explore new texts, diverse ways of thinking, and meaning making. Unfortunately, 
such engagement can also result in limited thinking processing with simplistic 
approaches and acceptance of unsubstantiated claims online. 

Educators can teach critical evaluation of sources and the skills students need, 
bringing into the spotlight the role of epistemic cognition — or thinking pro- 
cesses related to understanding the fundamental nature of knowledge and how 
we justify the truth. Dialogic pedagogies are particularly important to advance 
such cognition (Lunn et al., 2017). Some teachers, however, may over-estimate 
their students’ skills in navigating digital intertextual environments (Kirschner & 
De Bruyckere, 2017) as students, at all levels of education, often disregard the 
source of the information and only focus on the content of what they have read 
(Braten et al., 2019). 

Due to calls to advance students’ epistemic cognition (OECD, 2016), there 
has been increasing interest in approaches in schools (see Trevors et al., 2017). 
Source evaluation for critical reading is a task that requires cognitive attention 
to discriminate between trustworthy and untrustworthy sources, and informa- 
tion about individuals and organisations that create and publish content such 
as when, where, and for what purpose the content is created, as these details 
are essential to evaluating the credibility, reliability, and value of the texts. This 
requires a complex cycle of self-regulation and evaluation that students learn to 
orchestrate. 

As students seek out information about controversial issues (e.g. climate 
change, pandemics, health issues, mobile phone health risks and so on) from a 
variety of sources they will undoubtedly be presented with conflicting views 
and need to develop a critical-analytical stance towards reading information. 
Once they have established credible sources they will then compare, contrast, 
and make decisions from a critical perspective constructing knowledge from 
information. These decoding literacies raise challenges especially when they 
deal with ill-structured problems in the form of controversial issues (Yang & 
Tsai, 2010). 

As noted earlier, successful online readers engage in higher-order thinking 
processes — when judging information sources, monitoring their knowing processes, 
and regulating their alternative knowledge-seeking actions — that need to be 
taught. Students’ stances can be advanced by reflecting on issues to foster think- 
ing skills for engaging with socio-scientific issues online. Facilitating strategies to 
make informed decisions, as they analyse, synthesise, and evaluate varied sources 
of data and information, while understanding the complexity of connections 
inherent within contextualised socio-scientific issues (Zeidler et al., 2019) is 
critical. This approach involves student engagement by: 
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e utilising personally relevant, controversial, and ill-structured problems that 
require scientific, evidence-based reasoning to inform decisions about such 
topics; and 

e employing scientific topics with social ramifications that require students to 
engage in dialogue, discussion, debate, and argumentation (Zeidler, 2014 
p. 699). 


To advance such skills, Schiefer et al. (2020) implemented a science intervention 
to foster 7-8-year old’s epistemic beliefs. The ten-week program involved stu- 
dents actively participating in the scientific inquiry process and reflecting criti- 
cally on the epistemic issues that arose. While teaching science content was not 
the goal, related topics were used to illustrate aspects of science that provided 
opportunities to think critically in an inquiry-based environment. Students 
worked on topics already covered in their Year 3 and Year 4 classes (e.g. function- 
ing of the human senses; physical experiments about speed, research in a student 
neuroscience laboratory). 

Each science topic began with fully guided experiments and direct instruc- 
tion, followed by structured inquiry, and guided inquiry that then transitioned 
from hands-on activities to more complex processes of reflection and thinking 
about knowledge and knowing (Schiefer et al., 2020). Exercises and demonstra- 
tions (in pairs or with the whole group) were conducted with an active inquiry 
phase (in groups of two to three children). Reflection on results and inferences 
was supported through group activity, guided by the educator. Making time and 
space for these dialogic exchanges, deliberations about evidence, and personal 
interactions is critical (Lunn et al., 2017). 

Schiefer et al’s (2020) science intervention positively affected students’ epis- 
temic beliefs and their stances about the certainty, development, and justification 
of knowledge — with shifts towards an understanding that scientific knowledge is 
tentative and evolving rather than certain and fixed; complex and interconnected 
rather than piecemeal; justified by appeals to evidence and coherence rather than 
authority; and, constructed by people rather than perceived in nature (Elby 
et al., 2016). After facilitating such a program, students can then have meaningful 
discussions to engage in explorations into knowing and productively discuss their 
disagreements based on their research findings (Chinn et al., 2020). 

Drawing on evidence and sources of information students can engage in 
informed dialogic argumentation about alternative perspectives and justify their 
claims — leading to a better grasp of how scientific practices are evaluated and 
what kinds of arguments are relevant to evaluating them. Dialogic argumentation 
is an important avenue for developing competencies in identifying and weighing 
positive and negative attributes of conflicting perspectives on a particular issue, tak- 
ing relevant reasons and evidence for the different perspectives into consideration 
(Kuhn & Crowell, 2011). This is an essential feature of scientific reasoning — the 
ability to construct arguments that relate claims to evidence is also important in 
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informal contexts where people have to make rational judgements about contro- 
versial socio-scientific issues (Yang & Tsai, 2010). When coming across contra- 
dictory views on an issue, students should consider which claims can be justified 
as knowledge. 

Engaging in controversy, educators can teach disagreement as a genuine, unre- 
solved controversial issue on which even expert views diverge (Chinn et al., 
2020). This type of interaction between peers who hold an opposing position on 
a socio-scientific topic can support development of students’ skills as they produce 
two-sided rather than one-sided arguments (Crowell & Kuhn, 2014). As students 
become familiar with engagement in critical thinking, they can then move to 
sourcing further information online to integrate multiple texts on the same topic 
(Braten et al., 2019). This processing can be taught — for instance, to Josie who 
has been given the task to research mobile phone health risks — by scaffolding 
research on a topical issue as she comes to judge information sources, monitor her 
knowing processes, and regulate her alternative knowledge-seeking actions (Cho 
et al., 2018). As Josie goes about her task to report on health concerns related to 
mobile phones, she can be supported to move through a number of steps: 


¢ First, set goals based on the instructions, formulate questions, plan tasks, and 
map out a sequence to complete her report. 

e Then, assess the information she needs. This may include getting her to 
think about her epistemic stance (she may need to be supported to move 
beyond seeking one right answer, many right answers, to evaluate knowl- 
edge). Her epistemic stance will influence her personal online reading path- 
way, including initial sourcing. 

e In the next step, as she judges information her approach to knowledge can 
be supported to move from objectivist and/or subjectivist to an evalua- 
tivist stance. This will influence her consideration of the features of the 
sources such as trustworthiness of websites, credentials of the authors, and 
criteria for validity of information. Here, issues of confirmation bias can be 
addressed as Josie comes across information to support her own beliefs. 

e Next, Josie uses the resources to generate her report. Her approach to 
knowledge is important for monitoring as she interprets the content to check 
if she has met her goals based on her epistemic beliefs. Can she find evi- 
dence beyond one right answer or ‘truth’ (objectivist), different opinions 
that all seem to have merit (subjectivist), to evaluate the information based 
on criteria such the credibility of sources or evidence (evaluativist)? 

e In the final step, Josie reviews her report and is supported to present argu- 
ments and counter-arguments related to her topic and synthesise her 
findings. Entering into dialogic discussions with others at this stage will 
determine whether her goals have been achieved or whether she should 
re-cycle through previous processing steps to regulate alternative knowledge 
to achieve her goals. 
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Reflection on findings and inferences supported through dialogic interactions 
with peers and Josie’s teacher will help her develop justification of knowledge — with 
shifts towards understanding that scientific knowledge is tentative and evolving 
rather than certain and justified by evidence and coherence. 

Promising for educators, are positive relationships between beliefs in the need 
to evaluate and construct knowledge, learning on the internet and integrating 
sources related to socio-scientific tasks (Barzilai & Zohar, 2012). In this way, if 
young people believe in checking knowledge claims against other information 
sources, they can develop adaptive strategies to use for web-based inquiry learn- 
ing (Chiu et al., 2013). 


Critical issues in approaches to evaluation of socio-scientific 
issues 


Reading and evaluating sources of information on the internet related to socio- 
scientific issues can create tensions due to moral dimensions and individual dis- 
agreement. Societal dilemmas require advanced skills to evaluate information 
and moral reasoning skills that can be challenging for young people. A core 
‘post-truth’ challenge is how to overcome the prevalence of deep disagreements 
between individuals about ways of knowing (Chinn et al., 2020). Moral reason- 
ing involves the complex coordination of various beliefs (Scholes et al., 2021) as 
evidential reasoning (evaluating competing value claims and making a judgement 
based on the evaluation of such claims) that requires dialogic argumentation skills 
to effectively reason about a range of moral and social concerns (Reznitskaya & 
Wilkinson, 2017) 

The ethical considerations of many socio-scientific issues (such as genetic 
engineering involved in cloning and gene therapy) have moral dimensions. For 
young people to make informed decisions regarding socio-scientific issues, they 
have to consider the moral ramifications (Zeidler et al., 2019). Many ethical 
questions may be raised, for example, in relation to the right for parents to alter 
the genetic composition of their children, or whether the human genome should 
be subject to artificial manipulation. To reach a conclusion, individuals draw on 
moral and ethical dimensions of socio-scientific issues and establish efficacy of 
justifications. This is evident in other contemporary issues as well. 

Climate change is another such scientific issue and has been referred to as a 
‘perfect moral storm’ as it brings together major challenges to ethical action in 
a mutually reinforcing way (Gardiner, 2011). For instance, climate change is a 
global phenomenon, with inter-dependence between nations with different vul- 
nerabilities, inter-generational implications for decisions made now, and questions 
about the moral value of nonhuman nature, such as obligations to protect unique 
places. Facilitating strategies to make informed decisions, to analyse, synthesise, 
and evaluate varied sources of data related to socio-scientific issues often includes 
an ethical issue, adding complexity for young people (Zeidler et al., 2019). 
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At the same time, socio-scientific issues are contentious. Science is always 
open to new evidence. The great philosopher Karl Popper (2005) revolutionised 
contemporary thinking on science and knowledge, arguing that scientific knowl- 
edge is not the search for certainty as all human knowledge is fallible and there- 
fore inherently uncertain. This position offers a myriad of questions: How will 
new technologies influence what is known in the scientific realm or potentially 
challenge long-held facts? What new dilemmas will arise with the raft of technol- 
ogy coming on the market that may change science endeavours forever? While 
once a bionic eye was the mainstay of science fiction, in 2021, an Israeli surgeon 
implanted the world’s first artificial cornea into a bilaterally blind man (Solomon, 
2021). What new ethical dilemmas might result from such scientific advances? 

Personal approaches to evaluating knowledge, particularly related to socio- 
scientific issues, are complex and are also open to transition and change. As 
individuals move through life, their epistemological stance can change or be 
specific within a particular domain. Approaches to reasoning are understood 
to vary across a wide range of social contexts as individuals coordinate different 
forms of reasoning, values, and emotions with contextual and cultural variation 
in moral and non-moral social reasoning (Lunn et al., 2017). How best to pre- 
pare young people to critically evaluate multiple sources of information to equip 
them for their digital futures and for active participation and citizenship in the 
21st century is therefore debated. While there are clear advantages to develop- 
ing epistemically informed literacy skills, such approaches are relatively new, and 
scholars continue to discover understandings about best practices. 


Recommendations for research 


Advancing research related to decoding sources of socio-scientific texts online 
is not only a matter of improving student disciplinary knowledge on topics and 
academic achievement in science and English subjects, but also a matter of edu- 
cating for critical epistemic literacy. Eye-tracking software, for example, opens 
up opportunities for exploring more deeply how young people engage in inter- 
net searches as they explore hyperlinks, make decisions about personal reading 
pathways, and compare conflicting sources. Researchers can follow participants’ 
eyes during online tasks and get insight into the processes underlying behaviours 
to reveal patterns of inquiry, learning, and interactions. Such software can cap- 
ture this immersive exploration that results in exciting multimodal engagement 
and connections to things that were never possible. 

As students tend to assume that information on the internet is true and have 
to learn about the nature of the internet to become critically informed, more 
sophisticated understandings of students’ approaches as they engage in inquiry 
tasks can inform educational practice. Even good readers can spend an inordinate 
amount of time searching the internet and not be skilled at critically selecting 
online sources and developing reliable knowledge from information (Woodward & 
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Cho, 2020). We urgently require new knowledge to inform approaches to read- 
ing about controversial issues such as science-informed social problems. 

How to prepare teachers to advance their students’ thinking to engage in evi- 
dence-based evaluation of socio-scientific issues presents a gap in the research. 
Young elementary-aged children are more ready to advance epistemology than 
previously thought (Lunn et al., 2017; Schiefer et al., 2020). Teachers’ beliefs, par- 
ticularly their thinking about epistemic aims and reliable processes for achieving 
those aims may impact students’ understanding of complex, controversial issues 
(Braten et al., 2017). This is because teachers’ beliefs may facilitate or constrain 
their implementation of strategies aimed at engaging students in reasoned argu- 
mentation through classroom dialogue. Discussions of teacher knowledge rarely 
query educators about the knowledge that they need for teaching and few studies 
probe their beliefs about the nature of such knowledge (Fives & Buehl, 2010). To 
address this issue, further work is essential to examine the connections between 
teachers’ beliefs and the epistemic beliefs they foster in their students. This may 
require examination of how teachers reflect on their own beliefs in the context of 
dialogic-based instruction in order to calibrate it with the aim of deep understand- 
ing and the reliable process of reasoned argumentation for their students (Braten 
et al., 2017). 

More research, new data, and exploratory theories are also critical due to 
the rapid pace of evolving real-time information avenues and opportunities for 
learning. While algorithms and AI are part of everyday googling, what does 
the future of the Internet of Things (IoT) for example present? IoT integrates 
the interconnectedness of human culture with digital information systems — the 
internet — and young people will be able to have increasing access to endless 
information. Information, however, is not knowledge. What will be the role 
of reading in the digital future and how do we conceptualise evaluative critical 
reading in such a world? We argue decoding and evaluating conflicting texts 
online may need to become part of a new critical epistemic literacy paradigm. 


References 


Afflerbach, P, & Cho, B.Y. (2009). Determining and describing reading strategies: 
Internet and traditional forms of reading. In H.S. Waters, & W. Schneider (Eds.), 
Metacognition, strategy use, and instruction (pp. 201-225). Guilford. 

Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrat- 
ing online sources. Cognition and Instruction, 30(1), 39-85. https://doi.org/10.1080/ 
07370008.2011.636495 

Beach, R., Share, J., & Webb, A. (2017). Teaching climate change to adolescents: Reading, 
writing, and making a difference. Routledge. 

Bessi, A., Caldarelli, G., Del Vicario, M., Scala, A., & Quattrociocchi, W. (2014). Social 
determinants of content selection in the age of (mis) information. In L.M. Aiello, & 
D. McFarland (Eds.), Social informatics: 6th international conference, SocInfo 2014 
(pp. 259-268). Springer. 


Critically evaluating multiple sources for digital futures 61 


Bråten, I., Brante, E.W., & Strømsø, H.I. (2019). Teaching sourcing in upper secondary 
school: A comprehensive sourcing intervention with follow-up data. Reading Research 
Quarterly, 54(4), 481-505. https://doi.org/10.1002/rrq.253 

Bråten, I., Britt, M.A., Strømsø, H.I., & Rouet, J.F. (2011). The role of epistemic beliefs 
in the comprehension of multiple expository texts: Toward an integrated model. 
Educational Psychologist, 46(1), 48-70. https://doi.org/10.1080/00461520.2011. 
538647 

Braten, I., Muis, K.R., & Reznitskaya, A. (2017). Teachers’ epistemic cognition in the 
context of dialogic practice: A question of calibration? Educational Psychologist, 52(4), 
253-269. https://doi.org/10.1080/00461520.2017.1341319 

Bromme, R., Kienhues, D., & Stahl, E. (2008). Knowledge and epistemological beliefs: 
An intimate but complicate relationship. In M.S. Khine (Ed.), Knowing, knowledge and 
beliefs: Epistemological studies across diverse cultures (pp. 423-441). Springer. 

Castek, J., & Coiro, J. (2015). Understanding what students know: Evaluating their online 
research and reading comprehension skills. Journal of Adolescent & Adult Literacy, 58(7), 
546-549. https://doi.org/10.1002/jaal.402 

Chinn, C.A., Barzilai, S., & Duncan, R.G. (2020). Disagreeing about how to know: 
The instructional value of explorations into knowing. Educational Psychologist, 55(3), 
167—180. https://doi.org/10.1080/00461520.2020.1786387 

Chiu, Y.L., Liang, J.C., & Tsai, C.C. (2013). Internet-specific beliefs and self-regulated 
learning in online academic information searching. Metacognition and Learning, 8(3), 
235-260. https://doi.org/10.1007/s11409-013-9103-x 

Cho, B.Y., Woodward, L., & Li, D. (2018). Epistemic processing when adolescents read 
online: A verbal protocol analysis of more and less successful online readers. Reading 
Research Quarterly, 53(2), 197—221. https://doi.org/10.1002/rrq.190 

Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies 
used by sixth-grade skilled readers to search for and locate information on the Internet. 
Reading Research Quarterly, 42(2), 214-257. https://doi.org/10.1598/RRQ.42.2.2 

Crowell, A., & Kuhn, D. (2014). Developing dialogic argumentation skills: A 3-year 
intervention study. Journal of Cognition and Development, 15(2), 363-381. https://doi. 
org/10.1080/15248372.2012.725187 

Econsultancy. (2012). SEMPO state of search marketing report 2012. Available at 
https:// www.searchenginewatch.com/2012/09/13/sempo-north-american-sem- 
industry-to-reach-23-billion-in-2012/ 

Elby, A., Macrander, C., & Hammer, D. (2016). Epistemic cognition in science. In J.A. 
Greene, W.A. Sandoval & I. Braten (Eds.), Handbook of epistemic cognition (pp. 113-127). 
Routledge. 

Epstein, R., & Robertson, R.E. (2015, August). The search engine manipulation 
effect (SEME) and its possible impact on the outcomes of elections. Proceedings of 
the National Academy of Sciences, 112(33), E4512—E4521. https://doi.org/10.1073/ 
pnas.1419828112 

Erskine, E. (2019). New Zealand to include climate change, activism, and “eco-anxiety” 
curriculum in schools. One Green Planet. https://www.onegreenplanet.org/environ- 
ment/new-zealand-to-include-climate-change-activism-and-eco-anxiety-curricu- 
lum-in-schools/ 

Fang, Z., & Schleppegrell, MJ. (2010). Disciplinary literacies across content areas: 
Supporting secondary reading through functional language analysis. Journal of Adolescent 
& Adult Literacy, 53(7), 587-597. https://doi.org/10.1598/JAAL.53.7.6 


62 Mind and materiality 


Fives, H., & Buehl, M.M. (2010). Teachers’ articulation of beliefs about teach- 
ing knowledge: Conceptualizing a belief framework. In L.D. Bendixen, & FC. 
Feucht (Eds.), Personal epistemology in the classroom: Theory, research, and implications 
for practice (pp. 470-515). Cambridge University Press. https://doi.org/10.1017/ 
CBO9780511691904.015 

Furenes, M.I., Kucirkova, N., & Bus, A.G. (2021). A comparison of children’s reading on 
paper versus screen: A meta-analysis. Review of Educational Research, 91(4), 483-517. 
https://doi.org/10.3102/0034654321998074 

Gardiner, S.M. (2011). A perfect moral storm: The ethical challenge of climate change. Oxford 
University Press. 

Granka, L.A., Joachim, T., & Gay, G. (2004, July). Eye-tracking analysis of user behavior 
in www search. In Proceedings of the 27th Annual International Association for Computing 
Machinery Special Interest Group on Information Retrieval Conference on Research and Development 
in Information Retrieval (pp. 478-479). https://doi.org/10.1145/1008992.1009079 

Greene, J.A., Sandoval, W.A., & Braten, I. (2016). Handbook of epistemic cognition. 
Routledge. 

Hofer, B.K., & Pintrich, PR. (Eds.). (2002). Personal epistemology: The psychology of beliefs 
about knowledge and knowing. Routledge. 

Johnson, J. (2022, January 26). Global market share of search engines 2010—2021. Statista. 
https://www.statista.com/statistics/216573/worldwide-market-share-of-search- 
engines/ 

Kirschner, PA., & De Bruyckere, P. (2017). The myths of the digital native and the 
multitasker. Teaching and Teacher Education, 67, 135-142. https://doi.org/10.1016/j. 
tate.2017.06.001 

Kuhn, D., & Crowell, A. (2011). Dialogic argumentation as a vehicle for develop- 
ing young adolescents’ thinking. Psychological Science, 22(4), 545-552. https://doi. 
org/10.1177/095679761 1402512 

Kuhn, D., & Weinstock, M. (2002). What is epistemological thinking and why does 
it matter? In B.K. Hofer, & PR. Pintrich (Eds.), Personal epistemology: The psychol- 
ogy of beliefs about knowledge and knowing (pp. 121-144). Routledge. https://doi. 
org/10.4324/9780203424964 

Latour, B. (1987). Science in action: How to follow scientists and engineers through society. 
Harvard University Press. 

Lazer, D.M., Baum, M.A., Benkler, Y., Berinsky, A.J., Greenhill, K.M., Menczer, F, Metzger, 
MJ., Nyhan, B., Pennycook, G., Rothschild, D., Schudson, M., Sloman, S. A., Sunstein, 
C.R., Thorson, E.A., Watts, D.J., & Zittrain, J.L. (2018). The science of fake news. Science, 
359(6380), 1094-1096. https://doi.org/10.1126/science.aa02998 

Lunn, J., Johansson, E., Walker, S., & Scholes, L. (2017). Teaching for active citizenship: 
Moral values and personal epistemology in early years classrooms. Routledge. https://doi. 
org/10.1007/978-3-319-75559-5_5 

Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous 
reflections about knowledge and knowing during online information searching and 
their influence on learning. Learning and Instruction, 21(1), 137—151. https://doi. 
org/10.1016/j.learninstruc.2010.01.001 

Mason, L., & Scirica, F (2006). Prediction of students’ argumentation skills about contro- 
versial topics by epistemological understanding. Learning and Instruction, 16(5), 492-509. 
https://doi.org/10.1016/j.learninstruc.2006.09.007 


Critically evaluating multiple sources for digital futures 63 


Organisation for Economic Co-operation and Development (OECD) (2016). PISA 2015 
results in focus (Volume 1). https://doi.org/10.1787/9789264266490-en 

Popper, K. (2005). The logic of scientific discovery. Routledge. 

Reznitskaya, A. & Wilkinson, I. (2017). The most reasonable answer. Helping children build 
better arguments together. Harvard Education Press. 

Salmeron, L., Delgado, P., & Mason, L. (2020). Using eye-movement modelling examples 
to improve critical reading of multiple webpages on a conflicting topic. Journal of 
Computer Assisted Learning, 36(6), 1038-1051. https://doi.org/10.1111/jcal.12458 

Sandoval, W.A., Greene, J.A., & Bråten, I. (2016). Understanding and promoting 
thinking about knowledge: Origins, issues, and future directions of research on 
epistemic cognition. Review of Research in Education, 40(1), 457-496. https://doi. 
org/10.3102/0091732x16669319 

Schiefer, J., Golle, J., Tibus, M., Herbein, E., Gindele, V., Trautwein, U., & Oschatz, K. 
(2020). Effects of an extracurricular science intervention on elementary school children’s 
epistemic beliefs: A randomized controlled trial. British Journal of Educational Psychology, 
90(2), 382—402. https://doi.org/10.1111/bjep.12301 

Scholes, L., Wallace, E., Walker, S., Lunn Brownlee, J., & Lawson, V. (2021). Children’s 
epistemic reasoning about social inclusion of aggressive peers in a culturally diverse 
school. British Educational Research Journal, https://doi.org/10.1002/berj.3766 

Solomon, S. (2021, January 19). Haifa man regains his vision after getting artificial 
cornea implant. The Times of Israel. https://www.timesofisrael.com/haifa-resident- 
regains-vision-after-getting-artificial-cornea-implant/ 

Strømsø, H.I., & Bråten, I. (2014). Students’ sourcing while reading and writing from 
multiple web documents. Nordic Journal of Digital Literacy, 9(2), 92-111. https://doi. 
org/10.18261/ISSN1891-943X-2014-02-02 

Trevors, G.J., Muis, K.R., Pekrun, R., Sinatra, G.M., & Muijselaar, M.M.L. (2017). 
Exploring the relations between epistemic beliefs, emotions, and learning from 
texts. Contemporary Educational Psychology, 48, 116-132 https://doi.org/10.1016/j. 
cedpsych.2016.10.001 

Van der Linden, S., Leiserowitz, A., Rosenthal, S., & Maibach, E. (2017). Inoculating the 
public against misinformation about climate change. Global Challenges, 1(2), 1600008. 
https://doi.org/10.1002/gch2.201600008 

Woodward, L., & Cho, B.Y. (2020). How students’ beliefs about knowledge matter in 
multiple-source reading online: Implications for classroom instruction. Journal of 
Adolescent & Adult Literacy, 64(2), 135-144. https://doi.org/10.1002/jaal. 1062 

Yang, F-Y., & Tsai, C.C. (2010). Reasoning about science-related uncertain issues and 
epistemological perspectives among children. Instructional Science, 38(4), 325-354. 
https://doi.org/10.1007/s11251-008-9084-3 

Zeidler, D.L. (2014). Socioscientific issues as a curriculum emphasis: Theory, research, 
and practice. In N.G. Lederman, & S.K. Abell (Eds.), Handbook of research on science 
education, Volume II (pp. 697—726). Routledge. 

Zeidler, D.L., Herman, B.C., & Sadler, T.D. (2019). New directions in socioscientific 
issues research. Disciplinary and Interdisciplinary Science Education Research, 1(1), 1-9. 
https://doi.org/10.1186/s43031-019-0008-7 


4 


WHY VIDEO GAMING IS 
AN IMPORTANT DIGITAL 
LITERACY PRACTICE 


Amazing innovations in technology continue to advance video gaming expe- 
riences for children and young people around the world as they engage in 
immersive digital literacy practices. Each new generation of game adds to the 
excitement of the gaming experience, with ever-evolving high-fidelity graph- 
ics, facial and voice recognition, and new inclusions such as gesture control. 
Virtual reality (VR) headsets, wrap-around display screens, rooms augmented 
with wearable computers, and sensorial experiences related to scents and haptic 
devices are features games utilise to simulate real-world experiences. Advances in 
artificial intelligence (AI) generate sophisticated games that change and respond 
to player feedback, generating adaptive experiences that require cognitive flex- 
ibility and decision making by players. 

And what about the potentials of cloud gaming? Instead of creating video 
game systems that require more powerful hardware, developers spurred on by 
the potentials of massive cloud servers have developed technology that allows 
sophisticated images to be streamed to a screen through the internet. Children 
and young people with internet connections no longer need devices such as an 
Xbox to play games, as cloud gaming streams to a range of devices, such as 
smartphones, tablets, and PCs. Such advances offer higher accessibility and more 
opportunities for young people to engage in thrilling play. Many young people 
spend their time in such digital spaces at home, and increasingly in schools, 
whether it is play through mobile applications (e.g. Candy Crush), online multi- 
player sandbox modes (e.g. Minecraft), or virtual interaction with cultural places 
(e.g. Assassins Creed Odyssey Discovery Tour: Ancient Greece). Likewise, gamers 
can now engage in simulations of real-life experiences (e.g. The Sims, Real-Life), 
action games that emphasise physical challenges (e.g. Call of Duty), educational 
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games that promote coding (e.g. Roblox Scratch), or environments that merge 
construction genres and shooter games (e.g. Fortnite). 

Understanding the affordances of gaming for literacy education is important 
as the gaming industry shows no signs of slowing down — the total number of 
gamers is currently estimated at over three billion across the globe (Clement, 
2021). This is reflected in the number of Esports enthusiasts — multiplayer video 
games played competitively for spectators — that currently stands at 400 million, 
and growing (Gough, 2021). Young people make up a large percentage of Esports 
players with competitions based on video games growing rapidly in schools in 
the US, the UK, Australia, New Zealand, Asia, and beyond. For instance, the 
Federation of United Schools Esports — FUSE — Cup is a large international 
network of schools providing young students with an opportunity to participate 
in Esports competitions from the Asia Pacific Region (Australia, New Zealand, 
and Asia). There are three age divisions catering for children across the school 
years — Years 5 & 6, Years 7 & 8, and Years 9 & 10 (https://www.thefusecup. 
com.au/about-us). 

Individual schools are also getting in on the action and creating networks, with 
Kids in the Game bringing Esports to New York City middle schools by creating 
an inclusive league for 16 schools across the city (https://www.kidsinthegame. 
com). Esports is also moving into the main arena of play throughout the US, with 
Esports degrees available at several tertiary institutions including the University of 
Kentucky and The Ohio State University, while Marquette University launched 
the nation’s first Esports team at the highest level in college Esports in 2019 
(ManpowerGroup, 2021). 

Unprecedented numbers of children engaging in gaming offers the poten- 
tial to foster students’ critical literacy in virtual spaces (Qian & Clark, 2016; 
Scholes et al., 2021). From this perspective, immersive gaming literacy experi- 
ences facilitate embodied cognition and extend real-life time and space (Freina & 
Ott, 2015). Engagement in video game spaces not only develops literacy skills 
as players read, decode texts, and comprehend story lines, such spaces also offer 
opportunities for an epistemological shift in learning away from the acquisition 
of facts that are right or wrong, towards constructivist spaces of self-generated 
ideas that can be tested. As part of play, beliefs about knowledge come to the fore 
as gamers deliberate, comprehend, make decisions, and act in games (Gee, 2007). 
Decision making to solve problems demands critical thinking or sophisticated 
epistemology. These opportunities can develop and trial epistemic skills that 
support the development of 21st-century literacy skills (Qian & Clark, 2016). 
While dramatically different to traditional educational priorities, 21st-century 
skills include critical thinking for reasoning, systems thinking, computational 
evaluation, decision making, and problem solving (Binkley et al., 2012). 

Gaming skills relate to these 21st-century skills that are prioritised in the work- 
force. As advanced economies transition from manufacturing to information and 
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knowledge services, technology is transforming the nature of work with a focus 
on information sharing, teamwork, and innovation, with success measured by 
the skills of people enabled by technologies to use information to solve real- 
world issues. Video games have the potential to advance such problem-solving 
skills and cultivate the soft skills that are increasingly valuable as automation and 
machines perform more routine tasks, while making many careers obsolete in 
the future (Adachi & Willoughby, 2013; Mann et al., 2020). Such play improves 
critical thinking, creativity, emotional intelligence, and complex problem solving — 
soft skills that are hard to find and even harder to train (ManpowerGroup, 2021). 
An estimated 43% of employers are finding it difficult to teach the soft skills they 
desire, such as collaboration, communication, and the ability to learn skills that 
can potentially be developed through gaming (ManpowerGroup, 2021). Even 
the military is hiring gamers in their quest for soldiers who can assimilate infor- 
mation, react swiftly, and coordinate actions (Molloy, 2019). 

The skills developed through game play are highly valued in futures-focused 
schools, with qualities developed through gaming that are transferable to the 
workplace (ManpowerGroup, 2021). Games that require strategic thinking and 
problem solving can translate to real-life problems and solutions. The future 
calls for a transformation of teaching and pedagogy that supports learners in 
developing thinking that involves boundary crossing, inter-domain knowl- 
edge, and flexible cognition that can be readily applied to unfamiliar problems 
(Harris & de Bruin, 2018). These skills can be supported through teaching 
approaches that maximise students’ inquisitiveness to learn, immersive experi- 
ences that provide immediate feedback, and experiences embedded in real- 
world engagement (Gee, 2007). 

As the gaming industry continues its skyrocket in growth in the coming years, 
and young people engage in increasingly complex and demanding cognitive, 
linguistic, and socio-cultural practices generated by game play, the potential of 
video games for ‘hard fun’ or ‘serious play’ has important implications for edu- 
cation (Beavis, 2015). While connections between video games and text-based 
literacy learning have long been recognised in education, the skills children 
develop through in-play decision making in terms of critical literacy, potentials 
for developing the thinking skills needed in 21st-century classrooms across the 
curriculum, and preparation for modern work-life trajectories have not yet been 
fully understood. Re-calibrating an epistemological shift moves beyond young 
people’s acquisition of facts that are right or wrong. 

While traditional educational practices emphasise one right answer to a prob- 
lem (objectivist epistemology) within a framework of high-stakes testing rein- 
scribe conformity and standardisation (Plucker & Makel, 2010), video game 
play has the potential to advance digital literacies and provide opportunities for 
engagement in situated, active, problem-based learning environments. These vir- 
tual environments offer creative spaces for trialling alternative pathways and new 
innovations, or multiple right answers to a problem (subjectivist epistemology), 
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while working in teams or through individual endeavours. Players acquire prac- 
tices and ways of thinking that are innovative for trialling alternative approaches 
to real problems. In this sense, well-designed games are more than immersive 
virtual worlds that can accommodate a myriad of possibilities, where experien- 
tial learning allows participants to learn by doing, taking actions, accomplishing 
tasks, and by thinking and making decisions that have consequences. In this way, 
players engage with ways of knowing that impact on their experience — essentially 
learning to think in innovative ways. 

In the preceding chapter (Chapter 3), we looked at critical literacy skills for 
engaging in hyperlinked internet spaces and sourcing information on the World 
Wide Web. In this chapter, we consider some of the advances in video games 
that provide opportunities for critical literacy as players decode, evaluate, and 
make decisions in collective spaces of self-generated ideas. Drawing on gam- 
ing examples, we illustrate how educational change necessitates a reconceptu- 
alisation of knowledge goals that goes beyond the technological or pedagogical 
to envisage how video games can facilitate the epistemological. We argue for 
new conceptions of learning environments in education spaces that draw on 
and develop student expertise and digital literacies related to video gaming. This 
approach provides a response to future needs for digital knowledge creation and 
answers to key questions about how to foster and develop new knowledge and 
critical literacies for the future. 


Advances in video games and implications for epistemic thinking 


Big tech companies are taking advantage of gaming phenomena and investing 
heavily in the development of gaming products. While video games were once 
thought to be the domain of geeky males, expanded market demographics have 
made video games mainstream to the point where people are playing games 
earlier in life, and the gender binary is blurring. Emerging technologies that are 
taking video games to new heights provide increasing opportunities for players’ 
critical engagement as they assess the context, evaluate multiple options, work 
collaboratively, make decisions to solve problems, and constructively innovate 
and contribute new knowledge. 

Knowledge — or how players think about knowledge — is important for play- 
ing video games. Theorists propose that individuals start out thinking about 
knowledge in objective terms, where knowledge claims are either true or false 
(Kuhn et al., 2000; Kuhn & Weinstock, 2002). They may then shift to a sub- 
jectivist perspective, where knowledge is considered uncertain, given many pos- 
sible answers. Finally, some will become evaluativist, accepting knowledge as 
constructed and uncertain, but believing that knowledge claims can be evaluated 
against established criteria (see Chapter 3 for a review). 

Engagement as a player in most video games involves a range of tasks that 
are epistemologically demanding and require rational decision making under 
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uncertainty, pursuing the best course of action in light of one’s beliefs or infor- 
mation (Peterson, 2009). Avid players’ thinking processes become more efficient 
as they collect visual and auditory information to inform their decision mak- 
ing in the game. This process occurs much faster in gamers than non-gamers 
(Bavelier et al., 2011). Decisions are never black and white, as the brain is always 
computing probabilities that require evaluative critical thinking skills (Bavelier 
et al., 2011). In this way, experiences in gaming spaces can advance 21st-century 
skills required and necessary for educated citizens to think critically about com- 
plex, controversial issues (Greene & Yu, 2016). Video games offer opportunities 
for students to learn using techniques of innovation — ways of learning that facili- 
tate immersion in a practice (Shafter & Gee, 2005). These skills have application 
across the curriculum, which will be discussed in more depth throughout the 
chapter. 


Epistemic video games 


When thinking about gaming and epistemology, the first thing that comes to 
mind may well be ‘epistemic games’. This chapter, however, makes a clear dis- 
tinction between the broad epistemic skills players draw upon (beliefs about 
knowledge and knowing) as they engage in a wide variety of mainstream video 
games, and traditionally defined epistemic games — games that enculture players 
into a way of thinking about a problem that is particularly relevant to a specific 
profession. 

Traditional epistemic games are designed around a profession that has a cul- 
ture composed of skills, values, knowledge, identities, and an epistemology that 
anchors how professionals operate (Shaffer, 2009). Nurturing a player’s way of 
thinking in a defined context, these games teach skills that prepare individuals 
for the complex thought processes required in high-level professions, such as sur- 
gery, urban planning, and mechanical engineering. Opportunities are afforded 
for students to role-play professions and prepare for life outside of school. Players 
learn knowledge through action in specific contexts — as they make knowledge, 
apply knowledge, and share knowledge (Shaffer & Gee, 2005). 

Games that are epistemic in nature have been designed to be instructional. For 
instance, Nephrotex, developed by the departments of Biomedical Engineering, 
Engineering Physics, and Educational Psychology at the University of Wisconsin, 
was created for first-year engineering students to allow them to engage in a vir- 
tual internship in a fictitious biomedical engineering design firm. The students 
face real-world stakeholders, such as collaborating engineers, marketing and 
product representatives, and feedback from focus groups as they make choices 
and create their designs in the game (Chesler et al., 2013). 

Key to the design of epistemic games is the player’s unique ability to engage 
in the environment as an autonomous character, as the freedom to develop a 
character engages and promotes ownership of the environment (Annetta, 2008). 
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Devlin (2011), a mathematician who launched the company BrainQuake to 
create mathematics learning video games, aimed to develop players’ mathemati- 
cal thinking and ability to adopt the identity of a mathematically able person. 
This requires moving from skill-and-drill questions to involve the player in 
genuine mathematics problem solving within the game environment — moving 
into the epistemic realm. This skill is increasing in demand as young people 
and future workforces need the ability to take a novel problem that is not well- 
defined and does not have a single ‘right’ answer (objective epistemology), and 
make progress with it, while coming up with alternative answers (subjective 
epistemology). 

While epistemic games traditionally have a narrow definition, emerging main- 
stream games also include strategic play, with knowledge in complex domains 
based on the study of disciplinary communities, such as the social, physical, and 
biological sciences. While not designed as an epistemic game, Minecraft has been 
used in classrooms and with young people around the world to advance disci- 
plinary knowledge (Lane & Yi, 2017; Short, 2012). Minecraft is constructionist 
in its nature, offering a different style of instruction than is typically employed 
in many classrooms (Cipollone et al., 2014). It is a multiplayer sandbox video 
game, meaning there is no linear narrative structure to guide players. In a simi- 
lar manner to other sandbox games, such as The Sims (strategic life simulation 
video game), players achieve success as they experiment within the environment, 
individually or with multiple players. The Minecraft environment, however, is 
graphically very simple and encourages more interaction with naturalistic agrar- 
ian activities (e.g. mining coal, crafting shelters, or harvesting wheat) and urban 
environments (homes, buildings, cities, or public spaces). Like a virtual world of 
Lego, it invites players on a range of devices (see Figure 4.1) to create their own 
world of buildings, villages, and other spatial elements — lending itself to real- 
world applications in creative fields such as public space planning. 

Minecraft has been used by the Block by Block project (https://www. 
blockbyblock.org), initiated in Sweden in 2013, to develop urban workshops to 
include youth in urban design. For instance, this process was used to design the 
first skatepark in Kosovo, one of the poorest areas of Europe. Other commu- 
nity projects have included young people in designing public spaces. Examples 
include the designing of a dump site (Nairobi), a park design (Gautam Nagar, 
Mumbai), a Market Hall development (Mogadishu, Somalia), and a playground 
(Gaza) (see Figure 4.2). 

Using Minecraft to engage with communities across the world, individual 
projects engage young people who do not typically have a voice in public proj- 
ects. Block by Block provides the training, the tools, and the Minecraft platform 
to participate, contribute ideas, and develop innovations through a collaborative 
process that helps all participants expand their perspective. This approach was 
used in a community project in San Paulo, Brazil (see Figures 4.3a, 4.3b, 4.4a, 


and 4.4b). 
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FIGURE 4.1 Minecraft can be accessed on mobile devices 
Photo by Mika Baumeister/Unsplash 


Minecraft also provides a space for teams to try out ideas, present models, and 
advocate for their ideas to stakeholders and professionals including urban plan- 
ners, architects, and local policy makers. In this way, young people have oppor- 
tunities to think and act like urban planners, and to build the skills that are so 
important in 21st-century workplaces. 


Serious video games 


Perhaps you may also think about ‘serious games’ when considering the epis- 
temic nature of gaming. Similar to epistemic games, serious games are not 
designed for pure entertainment, but for training or skill development in indus- 
tries like education, health, medicine, science, the military, city planning, and 
engineering. One nuance between traditional epistemic and serious games is that 
the ‘serious’ application articulates into education, defence, aeronautics, science, 
or health for the purpose of training to teach skills. These skills may not be epis- 
temic related and may be drill-and-skill. They also tend to include fundamental 
elements related to game dynamics, such as rankings, rewards, badges, or points 
systems that are intended to motivate players. For instance, Duolingo helps users 
learn English, Spanish, French, or German as they receive points, go up to the 
next level, lose lives, or outdo their friends and relations. 
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FIGURE 4.2 Community design © UN-Habitat. (a) A playground in Gaza designed 
using Minecraft — before (b) A playground in Gaza designed using Minecraft — after 


(b) 


FIGURE 4.3 Community design © UN-Habitat. (a) Minecraft used in community 
urban design in Brazil — before. (b) Minecraft used in community urban design in 
Brazil — after 
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(b) 


FIGURE 4.4 Community design © UN-Habitat. (a) Minecraft used in community 
urban design in Brazil — in progress. (b) Minecraft used in community urban design in 
Brazil — final Minecraft design 


VR environments have been excluded from many educational settings due to 
the high cost of the equipment, with their usage over the past 50 years restricted 
to military applications and research institutes (Checa & Bustillo, 2020). Cost 
issues may also preclude their use in some mainstream school contexts where 
they could be most valuable. The launch of new high-quality affordable hard- 
ware and software media for VR since 2015, however, has boosted opportunities 
for such applications. 

While serious games with VR options can improve user experiences, and 
therefore knowledge acquisition, this environment is still an emerging field. 
New questions continue to arise about the best way to design efficient serious 
games for such environments (Checa & Bustillo, 2020). Many games, however, 
are already moving into the VR space, with Minecraft’s VR version enhanced for 
playing solo or with friends. The platform includes 3D audio, a virtual living 
room mode (for a break from the first-person perspective), VR Turning (head 
swivelling motions), and VR Controls (to make in-game tasks easier). 

New cutting-edge VR environments present highly learner-centred envi- 
ronments for serious game platforms that produce immersion and interactivity. 
These experiences can create situations that could not otherwise be experienced 
in real life, including ethical dilemmas, dangerous, and even impossible situa- 
tions in terms of time and space (Freina & Ott, 2015). Players feel in control of 
an interactive learning process that facilitates active and critical learning. Role- 
playing through VR has been used in therapeutic contexts, conflict mediation, 
restorative justice, and many other fields to help participants visualise events and 
conflicts from the perspective of others (Bertrand et al., 2018). VR supports 
multisensory and motor stimuli in synchronicity with the first-person perspec- 
tive of an avatar, so that players feel they have swapped bodies with another per- 
son through multisensory stimuli (Maselli & Slater, 2013). Manipulations of the 
senses can be used to modulate empathic responses with significant plasticity of 
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empathic abilities even decreasing implicit racial biases (Peck et al., 2013). The 
ability of immersive VR to displace the first-person point of view relates directly 
to perspective-taking from another point of view — potentially fostering the abil- 
ity to see alternative perspectives. 

Serious games have also been touted as an avenue for fostering empathy. 
Players have the simulated experience of walking in someone else’s shoes through 
the illusion of embodiment. Empathy, as an affective state, results as an interlay of 
multiple neural circuits related to motor, cognitive, emotional, motivational, and 
behavioural functions (McCall & Singer, 2013). Enhancing empathy in children 
has the potential for further developing social skills, such as interpersonal com- 
munication, problem-solving abilities, and emotional regulation, as well as other 
empathy-related phenomena, including perspective-taking. This allows children 
to better understand and learn from the actions of others (Bertrand et al., 2018). 
Role play in life simulation games, such as RealLives, can foster empathy because 
it allows players to inhabit the lives of individuals around the world. Students 
who play RealLives as part of their curriculum can potentially show more global 
empathy (observed in their identification with the characters played), and greater 
interest in learning about other countries (Bachen et al., 2012). 

RealLives provides a simulation, where players test themselves against the 
many unpredictable life conditions that occur in the course of a lifetime — from 
birth to death. You can choose your occupation, living conditions, social activi- 
ties, and start families, but all decisions can be affected by random events such 
as floods, outbreaks of war, disease, car accidents, and other major life-changing 
events. Players are given 11 attributes that include health, resilience, happiness, 
intelligence, artistic, musical, athletic, strength, endurance, spirituality, and wis- 
dom, but these change through events and decisions in the game. The scenarios 
that play out over the course of the player’s life are directly elicited from the data 
of the country in which the character resides, with tools available to learn about 
the culture, socio-economic conditions, and other metrics of the country of the 
character’s birth in the game. This also provides a realistic narrative of various 
cultures, and political and economic systems. RealLives play allows individuals to 
try out options where there is no one right answer (objectivist epistemology), 
but multiple right options (subjectivist epistemology), with decisions that flow 
into different trajectories with associated consequences. 

Many games also focus on physical or virtual interaction with a cultural or 
historical place and its objects. Games related to cultural heritage issues that 
are used in formal education still have many challenges, particularly related to 
effective storytelling and the evaluation of the effect on student learning per- 
formance (Malegiannaki & Daradoumis, 2017). The Discovery Tour game series 
that includes Assassins Creed Odyssey Discovery Tour: Ancient Greece, released in 
late 2018, has made inroads in presenting a game designed with a lavish bud- 
get and informed by historical research. In Assassin’s Creed Odyssey, you can 
choose to play either as a male (Alexios) or a female (Kassandra) — two siblings 
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separated from their Spartan parents during childhood. When choosing to play 
as Kassandra, players take on the role of a young, ambitious, vigorous, frustrated 
character living on a remote island. As she sets out to search for her real parents, 
her memories suggest a traumatic separation and many unanswered questions. 
She knows how to fight; however, she also enjoys exploring the world and creat- 
ing new friendships and possesses a sense of humour. The writers have succeeded 
in creating a character that players connect to, because Kassandra also possesses 
arguably human flaws, including naiveté, anger, and social awkwardness. 

While playing Assassin’s Creed Odyssey, gamers travel virtually to over 300 edu- 
cational stations found throughout the map and learn history through characters 
portrayed in the Odyssey games, including the Spartan warrior king Leonidas, 
the historian Herodotus, and Barnabas, who served as a naval captain during the 
Peloponnesian War. Assassin’s Creed Valhalla, launched in 2020, similarly offers 
an action role-playing experience for players, but this time the story is based on 
the Viking expansions into the British Isles. One of the many lessons that can 
be learned through these games is to listen when someone says something in the 
game that is important to them, no matter how difficult it is to perceive from the 
player’s personal position or point of view. Learning about subjectivist position- 
ing is also an element of multi-player games. 


Action and multi-player video games 


Action and multi-player games, by their very nature, also have epistemological 
implications. 

To be a successful multi-player, gamers need complex skills to select the best 
course of action in light of one’s beliefs and the information available to them 
(Peterson, 2009). The consequences of one’s decision depend on the choices 
of the other players involved in the situation and the context and are interre- 
lated with pathway choices. In this way, while not always intentionally planned, 
many mainstream popular video games provide rich spaces for developing and 
trying out epistemic skills. Playing video games is associated with improved 
multi-tasking, memory, attentional control, critical thinking, and problem- 
solving strategies (Qian & Clark, 2016). Action games also promote creativity 
and the ability to develop new cognitive learning templates (learning to learn), 
due to enhanced perception, attention, and cognition to become better players 
(Bejjanki et al., 2014). 

The potential benefits of video gaming are increasingly being documented, 
particularly in terms of the skills young people learn that translate to educa- 
tional domains and workplace skills for the future (Muriel & Crawford, 2018). 
Advances in action games offer opportunities for fostering decision making and 
problem solving, and produce improved cognition related to critical thinking 
(Powers et al., 2013). Some have argued that action video game players can 
become more efficient collectors of visual and auditory information, and they 
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therefore arrive at necessary thresholds of information they need in decision 
making much faster than non-gamers (Bavelier et al., 2011). 

Playing action video games such as Fortnite trains young people to make deci- 
sions faster. Decision making is crucial to success as every scenario is unique, and 
there are endless decisions to be made during a match. There are even YouTube 
clips on how to improve decision making in the game. Tips include reviewing 
decisions and watching replay highlights to evaluate what could have been done 
better. Just like most things in life, decision making has consequences and play- 
ers can learn through the experience of past mistakes. Improvement takes place 
as players develop a heightened sensitivity to what is going on around them, 
advancing their playing skills (Bavelier et al., 2011). Fortnite is not just a video- 
game shooter — it also has a creative sandbox mode in which users can create their 
own custom maps and game modes. This mode has the potential to encourage 
creativity and teach problem solving and engineering, similar to Minecraft. 

Action game play substantially improves performance in a range of attentional, 
perceptual, and cognitive tasks (Bejjanki et al., 2014). While many multi-player 
games have a strong competitive focus, there is also considerable co-operation 
between players that can be understood only in the context of the situation. 
For instance, Fortnite is often played in duos, four-player “Playground Mode’, 
or in teams, where the casual and social nature of this game reinforces existing 
social relationships and provides opportunities to learn teamwork, collaboration, 
strategic thinking, spatial understanding, and imagination (Carter et al., 2020). 
In multi-player games, knowledge and information are shared between players 
using digital game paratexts and online communities (Apperley & Beavis, 2013). 
This knowledge exchange may be facilitated by co-operation, or it may involve 
a more direct mentorship, as a more experienced player leads another through a 
difficult part of the game. It may also involve the need to evaluate the best course 
of action given the available information (Peterson, 2009). 

Regular action players also demonstrate perceptual templates better tuned 
to the task as they focus and centre attention (Beyanki et al., 2014). Increased 
attention can aid faster visual search rate, a reduction in the size of the attentional 
blink, better change detection, and an increase in the number of items that can 
be simultaneously tracked. This attention control can also lead to improved per- 
formance in high-level cognitive tasks, such as mental rotation and multi-tasking, 
with benefits carrying over to real-world domains, given that pilots and lapa- 
roscopic surgeons have been shown to outperform their peers after fast-paced, 
action-packed video game training (Chiappe et al., 2013). 

It may be that action games change the brains of players. Regular action 
players appear to be able to suppress distracting information better, allowing 
for increased focus on the goal at hand (Mishra et al., 2011). And they may be 
more efficient at directing attention during a demanding task, requiring atten- 
tion networks to work less hard to perform at the same level of ability (Bavelier 
et al., 2011). 
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Now that we have looked at some specific examples of games that promote 
epistemic thinking, we turn to how these games can be utilised to promote both 
content knowledge and soft skills in classrooms. 


Video games for advancing critical literacy in 
educational contexts 


The learning benefits of playing action video games as students make deci- 
sions and exert cognitive control to focus attention on a task goal are promising 
for education (Bavelier et al., 2011; Prensky, 2010). The metacognitive skill of 
learning to learn and cognitive flexibility to navigate a new learning goal also 
have important applications in the classroom. We argue that the opportunities 
to evaluate information and make informed decisions in game play also have 
implications for advancing epistemic skills. Cultivating critical literacy skills for 
classroom and real-world learning allows students to try out and advance their 
epistemic positioning by taking on different perspectives and evaluating different 
action options. 

Early theorising explored the potential affordances of video games in school- 
based learning environments within the context of new technologies (Beavis, 
1998; Gee, 2007; Ito et al., 2009; Jenkins, 2009; Prensky, 2010). Arguments 
were made that video games, as texts, belonged in the literacy classroom and 
had a place in educational endeavour in terms of opportunities to explore the 
construction of values and identities, the transformation of reading practices 
and new literacy practices, and abilities required by a digital world, building 
connections between in-school and out-of-school worlds (Beavis, 1998). With 
increased game play among children and young people, researchers have focused 
particularly on game-based learning (Gee, 2007), making links to educational 
imperatives such as media and literacy learning (Beavis, 2015; Dezuanni et al., 
2015; Gee, 2007), and mathematics and scientific concepts (Lane & Yi, 2017). 
Some researchers have also provided specific examples of the benefits of serious 
games (Jones et al., 2020; Scholes et al., 2014), and games for use in education 
contexts (Dezuanni et al., 2015). 

Video games that support real-world problem solving can complement 
school curriculum across the curriculum. The cubic geometry of a game such 
as Minecraft lends itself to the teaching of various academic subjects, as it has a 
functioning ecology, with chemistry and physics aspects interwoven within the 
game that can be used to develop the scientific literacy of players (Short, 2012). 
There are increasing illustrations of how Minecraft can facilitate problem-solving, 
self-direction, and collaboration skills for communicating scientific concepts and 
learning STEM skills like engineering (Lane & Yi, 2017). 

Minecraft game play also promotes learning of mathematical concepts (e.g. 
dividing supplies evenly among players; estimating the area needed to build a 
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city), historical concepts (e.g. famous buildings and landmarks), and STEM 
learning through modding and hacking — altering the original programming 
code of the game to enhance the game (Dezuanni et al., 2015; Lane & Yi, 2017). 
It is likely that playing Minecraft also offers students their first meaningful expo- 
sure to powerful fields in STEM, such as engineering, agriculture, and biology 
(Lane & Yi, 2017). 

The conceptualisation of games-as-text provides a mode of connecting digital 
games and children’s actions inside the game to the wider context in which they 
are situated, such as the classroom, out-of-school experiences, and world events 
(Apperley & Beavis, 2013). Our interest, however, is in epistemic affordances, 
and we are informed by Toh and Lim (2021), who recently built on Apperley 
and Beavis’s (2013) model of games-as-text. Taking the original model further, 
Toh and Lim (2021) focus on critical playing and learning, proposing a metalan- 
guage for digital play. Of interest from an epistemic stance, is their consideration 
of perspective — focalisation and shift. 

Internal focalisation on characters’ perspectives. Games such as The Sims (life 
simulation) and Assassin’s Creed (discovery tour game series) convey the expe- 
rience of subjectivity for the player, with the design presenting a diversity of 
perspectives including different ways of looking at and understanding virtual 
environments, narratives, and characters (Allison, 2015). This diversity of per- 
spectives allows players to access modes of thinking that accord with a perspec- 
tive other than their own. Internal focalisation occurs as players have insights 
into the characters as players are told only what a given character knows. In The 
Sims, this includes how hungry the characters are and what they are thinking. 
Such experiences potentially advance one’s epistemic stance as players move from 
experiencing an objectivist position (one right perspective), to discovering a 
subjectivist position (experiences of many possible perspectives). 

Perspective shifts. As players take on the roles of different characters, they 
need to shift perspective. This shift is important for understanding the char- 
acter in a narrative as the player projects the character’s emotional, cognitive, 
and behavioural states. In games with multiple characters, the player’s agency 
in the narrative gives them partial control to switch perspectives between dif- 
ferent characters to piece together a character’s story and to explore multiple 
approaches to progress the game. For example, in The Sims, the player controls 
one or more avatars that have wants and needs with qualities, personality traits, 
and dynamic statuses with other characters that influence their behaviours. As 
the player empathises with the character, they are motivated to provide for their 
needs. Players are privy to their character’s thinking and are, therefore, afforded 
opportunities for experiencing different perspectives. 

When engaged through a VR platform, such perspective-taking experi- 
ences can induce illusions of ownership over a virtual body as a fully immer- 
sive embodied experience (Peck et al., 2013). Multisensory feedback such as 
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visuomotor synchrony may heighten this illusion, as virtual bodies may have par- 
ticular ‘semantics’ associated with them — through normative beliefs and stereo- 
typing. Peck et al. (2013) show how VR embodiment of light-skinned people in 
a dark-skinned body can lead to a comparative reduction in their implicit racial 
bias, suggesting new ways to address deep-seated issues, such as racial bias. 

Serious games have the additional challenge over ‘non-serious’ commercial 
games to include learning opportunities as integral to the game play and story. 
It is critical that the game concept includes meticulous planning of each ele- 
ment of the game and integrating of elements so that a player’s state of flow 
during game play is retained (Csikszentmihalyi, 1998). They need to include 
exciting graphics, storylines, instant feedback, creative haptics, and a great hook 
(Csikszentmihalyi, 1998) within richly designed spaces to help players solve a 
problem, and model environments, behaviours, and concepts that allow them to 
be led from concreteness to the abstract (Gee, 2007). When video games are well 
designed, they provide learning paradigms that evoke a sense of pleasure for the 
player, thereby facilitating intrinsic motivation and allowing the player to learn in 
order to achieve mastery. Games that facilitate productive learning provide spaces 
for players to have immersive experiences and receive immediate feedback with 
opportunities to try again, opportunities for players’ scaffolding, opportunities to 
learn from the experiences of other players, and opportunities to learn about real 
life experiences (Gee, 2007). 

Let’s return to the video game Minecraft, and the two options for game play. 
In survival mode, the player begins resourceless and alone, but then has the 
opportunity to create a world by collecting resources, building structures, bat- 
tling mobs, eating, and exploring the world in an effort to survive and then 
thrive. Players can craft tools including swords and axes from materials such as 
wood, gold, stone, iron, and diamond. This then allows them to harvest crops 
including wheat for bread and build houses or other structures to stay the night 
and to survive menacing threats like the Ender Dragon. Survival mode is a goal- 
oriented mode, and to be successful the player aims to thrive despite perceived 
threats. 

The alternative game play option — creative mode — requires players to dem- 
onstrate creativity and skill in the way they chose to survive. There are elements 
of survival, creation, and multi-player collaboration that intentionally legitimise 
and rely upon the contributions of its player community in ways unlike many 
other games (Cipollone et al., 2014). Communities of players engage in practice 
that includes tutorials, modifications, communal servers for multi-player engage- 
ment, and creations that have incorporated Minecraft-based recreation of popular 
culture such as a scale replica of the Starship Enterprise and the Hogwarts School 
from Harry Potter (Cipollone et al., 2014). 

In recent years, a modification called Minecraft Education Edition was 
created specifically for use in educational settings. This potentially extends 
Minecraft communities of players into schools where play is based on creative 
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prowess, from the resourcefulness inside game world spaces to the creative 
modifications that players make, signified by the free exchange of users’ cre- 
ations (Cipollone et al., 2014). Here we not only see opportunities for con- 
structing knowledge, but also creating spaces for the generation of new ideas, 
innovations in thinking, and fostering the development of new knowledge for 
the future. 


Critical issues, tensions, and debates 


The potential relationship between violent video games and adolescent aggres- 
sion is an ongoing contestable topic. Many parents and educators worry about 
the suitability of video games for their children and their relevance to learn- 
ing. For over three decades, research has explored the possible links between 
video games and negative outcomes, including aggression, addiction, wellbe- 
ing, and cognitive functioning, with little consensus (Johannes et al., 2021). 
Furthermore, tensions have arisen because the reliability and reproducibility of 
valid studies are few and far between (Drummond et al., 2020). More recently 
there has been a shift from concerns about violent video games and aggres- 
sion to concerns about the association between the amount, or nature, of 
the time people spend playing video games and their wellbeing (Przybylski & 
Weinstein, 2019). 

The launch of the game Fortnite also heightened debates about childhood 
gaming addiction. There is limited research on the impact of game addiction 
on children, while there are calls for media debates around ‘problematic play’ to 
incorporate and be inclusive of the child’s right to play, and the relevance of play 
for children’s critical media literacies (Carter et al., 2020). While controversy will 
continue, recent research suggests video game play can be an activity that relates 
positively to people’s mental health and, as such, regulating games could with- 
hold those benefits from players (Johannes et al., 2021). That is, video game play 
has been found to generate positive affect and social functioning, contributing 
to and supporting mental health and wellbeing (Jones et al., 2014). That’s right — 
playing video games can be good for your wellbeing and mental health (Gee, 
2007; Johannes et al., 2021; Jones et al., 2014). Nonetheless, there are clearly 
ongoing concerns that impact on the take-up of video games in education, cre- 
ating tensions for many educators who wish to embrace the positive learning 
experiences for their students through video games. 

There are also broader issues. The advent and the global success of Fortnite 
is increasingly linked to school curriculums through modes such as Fortnite 
Creative — where students build their own world within the game, encouraging 
computational thinking and problem solving. However, its use raises issues of 
consent (13 years and older), privacy (voice-chat and text-chat capabilities), and 
appropriateness (violence). While these issues will be ongoing, discourse related 
to the value of games for learning is still critical (Scholes et al., 2021). 
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Recommendations for research 


Many consider gaming a powerful and exciting medium for engagement and 
learning in the classroom (Ito et al., 2009; Prensky, 2010). Video games in edu- 
cational contexts can facilitate motivation, cognitive development, higher-order 
thinking, literacy learning, problem solving, decision making, multi-tasking, and 
collaboration (Gee, 2007). Serious games are often used in education to teach 
and improve concepts, including mathematics (e.g. Mathletics), and reading (e.g. 
Teach your Monster to Read). Such games, however, are often based on players 
progressing through drill questions such as mathematics games that quiz play- 
ers on their times tables. Games designed to embed traditional curriculum can 
then lose the motivational, collaborative, engaging, and novelty aspects of the 
digital experience. When games emphasise external rewards and reinforcement 
for very school-like tasks, and educational content is forced into the video game 
medium, young people can quickly turn off (Ito, 2008). 

While the efficiency of video games as learning tools has been applied to seri- 
ous games or games-based learning as in disciplines, the importance of broader 
video games for epistemic literacy across the curriculum needs more research 
to make the educational application visible for educators. This is particularly 
true for educational change that necessitates a reconceptualisation of knowl- 
edge beyond the technological or pedagogical to envisage how video games 
can facilitate the epistemological. Ultimately, good games provide well-designed 
experiences in problem solving (Gee, 2007). More in-school research is needed, 
however, on how to implement games-based experiences in decision making 
to advance student epistemic skills and critical thinking. While VR environ- 
ments have been excluded from educational settings, due to the high cost of 
VR equipment, new research opportunities are now available as schools move 
towards investing in more affordable opportunities for students to learn through 
immersive VR environments. 
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PART II 
Body and senses 


The materiality of textual practices is continually changing as the substrates of 
reading and writing extend beyond paper to include a wide array of digital 
displays, from mobile devices to wearable technologies. The digitalisation of 
communication has clearly augmented the way texts are created, reproduced, 
and distributed. New fabrication technologies have transformed outputs for 
computer-mediated textual production that were previously limited to screen 
or coloured printers. Digital writing and graphic displays can be produced as 
physical inscriptions on almost anything — from fabric to vinyl — while texts can 
be tangible objects that are lathed, lasered, moulded, or extruded using 3D print- 
ing, rapid prototyping, 3D pens, computer-controlled routing, or laser cutting. 
Contrastingly, digital texts can be completely lacking material substance, such 
as virtual, augmented, and mixed reality objects that are overlayed upon the real 
world. 

Amid the changing materiality of the textual environment, a renewed focus 
on embodiment in the following section of this book — Body and Senses — draws 
attention to specific and vital ways in which the body mediates the work of 
the mind and the material world of textual practices. Of particular importance, 
haptic perception and proprioception — the movement, position, and sense of a 
body’s place in the world — are critical to literacy practices (Haas & McGrath, 
2018). What is important here is that the sensorimotor interaction of reading 
and writing with the material tools of textual practices are inextricably tied to 
cognition. For example, different regions of the brain are active when a writer 
forms letters using handwriting compared to typing with a keyboard (Mangen & 
Velay, 2010). With each new technology for inscription comes new questions 
about how different proprioceptive actions of the body shape the mind, and in 
turn, how these actions shape the use of technologies for encoding and decoding. 
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The section extends a sensory approach to understanding the body and the 
mind in literacy practices, an approach that was first developed by Mills (2016) to 
move from ocularcentric accounts of multimodality in textual practices to better 
account for the full sensorium. The recognition of embodiment in digital media 
practice is not simply to advocate that more movement, or larger body move- 
ment, is necessarily better for literacy practice. Rather, this paradigm involves a 
change in thinking from focusing exclusively on the cognitive, grammatical, or 
social dimensions of literacy practices, to take into account the material interac- 
tions with physical technologies that are mediated by the body in specific ways. 
We know that grammatical structures too are not arbitrary, attributable only to 
conventions, but are based on embodied experiences and image schemas that are 
established through interaction with the environment and often metaphorically 
extended (Gibbs, 2005). A sensory approach has new implications for under- 
standing how readers process what they read, and for how writers engage with 
the materiality of texts. 

Chapter 5 begins with a focus on sonic literacies associated with hearing and 
vocalisation, and then moves to consider the so-called ‘lesser’ senses of smell and 
taste in digitally mediated communication practices. The role of sonic experi- 
ence in digital media is central in the body-mind-world nexus of new literacies, 
whether of recorded speech, music, sound effects, or silence. Likewise, olfactory- 
based multi-sensorial media are becoming more prominent in film and cinema, 
virtual reality technology, gaming, and alert systems (Murray et al., 2016). There 
are yet unexplored relationships between olfaction and remembering the past, 
smell and social interaction, and between smell and other forms of language, 
such as writing. Taste or gustation, which is seldom theorised in literacy stud- 
ies, is considered, a sensory modality sometimes deemed a final frontier in the 
development of media simulations. Evidence is presented to demonstrate the 
connections between gustation and literary experiences, and new directions for 
gustatory output devices that may shape digital communication in the future. 
Finally, crossmodal relationships between various body-based modalities are 
examined, such as the effects of odours as authors compose texts, or the crossmo- 
dal meaning making that occurs when listening to music while tasting different 
flavours. 

Chapter 6 unravels the key role of two main kinaesthetic and propriocep- 
tive bodily engagements in literacy practices — first, haptics or touch; and sec- 
ond, locomotion or movement of the feet. It presents convincing arguments to 
demonstrate how sensorimotor experiences form a critical part of language and 
thought from birth and continuing throughout the life course. It critically delves 
into an array of new media that have opened fundamentally different opportuni- 
ties for understanding and optimising body motion and locomotion for literacy 
learning. 

Chapter 7 brings together the research on what is collectively called extended 
reality (XR), which includes virtual reality (VR), augmented reality (AR), and 
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mixed reality (MR) technologies, as well as innovations still in development 
(Mills, 2022). The chapter provides what may be the first systematic framework 
for theorising the material and social facets of extended reality that condition 
the nature and interactive scope of these new literacies. It critically examines the 
tensions and constraints for literacy education, while envisaging the potentials for 
extended reality technologies in curriculum and pedagogy. 

The common thread binding the chapters in this section is the relations 
between the materiality of media and the bodily engagement of text users, and 
their implications for embodied cognition. A central focus is an appreciation that 
the work of the mind in new literacy practices depends on embodied action 
with digital texts, whose materiality is undeniably being radically refashioned. 
The aim is to inspire new ways of conceptualising and systematising these media 
practices by taking account of the tangible substrates and implements of literacy, 
or conversely, the virtual immateriality of texts (such as mixed reality holograms), 
in digital worlds of language and social interaction. 
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5 


EMBODIMENT, LITERACIES, AND 
DIGITAL MEDIA 


This chapter argues for the continued development of a sensory literacies 
approach to revitalise thinking about the changing materiality of digital media 
and literacy practices, and relatedly, the transformed role of the body and mind 
in hybrid communication. A sensory literacies approach was developed by Mills 
(2016) in Literacy Theories for the Digital Age, building upon related research by 
Mills and colleagues (Friend & Mills, 2021; Mills & Exley, 2022; Mills et al., 
2018). It is an approach that is supported by a growing body of evidence from 
research in embodied cognition that looks specifically at the effects of diverse 
kinds of human understanding on the performance of language and communica- 
tion (Gibbs, 2005; Skulmowski & Rey, 2018). 

There has been a renewed focus on the role of the senses and embodiment 
across a range of disciplines: sensory studies in anthropology and sensuous schol- 
arship (Stoller, 1997), sensory research methodologies (Pink, 2015; Warren, 
2008), sensory sociology (Simmel, 1997), architectural sensoriality (Pallasmaa, 
2005), filmic sensoriality (MacDougall, 2005), culture and the senses (Classen, 
1999), sensory marketing, sensuous geography (Rodaway, 2002), and many oth- 
ers. Yet many cognitive scientists have approached research of cognition and 
meaning making as an abstract process involving disembodied symbols, divorced 
from bodily experience (Gibbs, 2005). Conventional perspectives of language 
learning that are centred on its neural basis have given insufficient attention 
to the way in which bodily activity from birth — activity which continues to 
develop in specialised and expert ways throughout the life course — is based 
on real-world experience and referents, and ordinary kinaesthetic interactions 
in a mind-body—environment relationship (Gibbs, 2005; Mills & Exley, 2022). 
Likewise, the influence of embodied cognition on literacy theories, such as new 
literacy studies, multimodal literacy, and other perspectives of digital media 
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production, has only recently begun to take centre stage (Ehret & Hollett, 2014). 
Others have noted a conspicuous gap in socio-cultural literacy research on mate- 
riality and the sensorimotor aspects of literacy practice (Mangen & Velay, 2010). 

A sensory literacies approach foregrounds the role of sensorimotor processes as 
fundamental to language and literacies, both digital and non-digital. At the same 
time, the rapidly changing ways to engage in digital media practices and com- 
munication have opened a broadened range of bodily ways of interacting with 
information and media. For example, since 2008, the world has seen the public 
release of technologies such as touchscreen iPads, smartwatches, fitness track- 
ers, Google Chrome, Snapchat, Pinterest, Instagram, TikTok, Google Drive, 
GPS from mobile phones, cryptocurrency and blockchain, wireless earphones, 
touchscreen gloves, Oculus Rift and HTC Vive virtual reality systems, Microsoft 
HoloLens (mixed reality), Google Glass, Apple Pencil, Amazon’s virtual assistant 
Alexa, and widely known games, like Pokémon GO, Minecraft, and Fortnite. 

Many of these technology platforms, social media sites, games, and appli- 
cations involve the user in varieties of bodily interactions that have specific 
influences on everyday language and information use. A sensory approach to 
literacies acknowledges the inter-relationships between the mind, the body, and 
multisensory digital environments that can require the use of vision, hearing, 
touch, locomotion, olfaction, proprioception (body position awareness), equi- 
librioception (balance), and more, as users interact with corporeal and virtual 
technologies and materials (Mills & Exley, 2022). 

The recognition of embodiment in digital media practice is not simply to 
advocate that more movement, or larger body movement, is necessarily better for 
literacy practice than less movement, or practices that involve only fine-motor 
movement. Rather, literacy enactments involve at least two key dimensions that 
can be used to classify the varieties of embodied literacy practices: (i) The degree 
or level of bodily engagement and (ii) the meaningfulness of bodily actions in 
relation to the specific literacy practice (Skulmowski & Rey, 2018). For example, 
tracing letters with fingers involves a small degree of movement and a high 
degree of meaningful task integration that can improve children’s learning of let- 
ter recognition and handwriting letter formation (Brookes & Goldin-Meadow, 
2016). Contrastingly, locomotion or walking while verbally responding to ques- 
tions involves exceptionally large body movements, and produces increased cre- 
ativity, compared to being seated (Oppezzo & Schwartz, 2014). Both large and 
small embodied actions that are meaningfully integrated can support literacy and 
digital media practices. 

The meaningfulness or degree of task integration of embodied action in 
literacy practices is illustrated in a study of children’s foreign language learn- 
ing (Mavilidi et al., 2015). The researchers looked at how children enacted the 
meanings of foreign language words in bodily ways that were relevant to the word 
meanings. For example, for the word ‘fly’, children would extend their arms out 
to the side of their bodies as if flying like a bird. They compared this to the 
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introduction of foreign language words while children performed an unrelated 
physical activity, such as running or walking for all words, with no relation to the 
meaning. Not surprisingly, the relevant task-integrated actions resulted in higher 
performance in foreign language learning than unrelated physical movement. 

Our key argument then is not that more extensive sensorial activity is vital 
to all language and digital media practices, but rather we need to give higher 
priority to understanding the specific ways in which the body is related to the 
performance of new literacy practices. Specifically, embodied literacies can be 
considered in terms of the varied extent of bodily action (e.g. using game con- 
troller, gestures when presenting, walking in virtual reality gaming), in addition 
to the degree of meaningful body—mind—language integration (Skulmowski & 
Rey, 2018). 


Marginalised senses in embodied media literacies: Hearing, 
smell, and taste 


New bodily and sensorial relations are being reconfigured in literacies and digital 
media practices that are changing much faster than in the previous centuries. 
This chapter considers the importance of sensorial and bodily interactions that 
matter for literacy learning in a contemporary world, with examples of digital 
and non-digital media practices. The examples in this section introduce hearing, 
and the so-called ‘lesser’ senses of smell and taste, and in the next chapter, haptics, 
and locomotion (see Chapter 6, this volume). Many of the visual meanings of 
digital texts are addressed in Chapters 8-10. The current chapter also explores 
the relations between the changing materiality of technologies and their rela- 
tionship to digital media practices and embodied cognition. 


Hearing: Sonic dimensions of language, literacy, and digital media 


The digital production and distribution of sonic media is a rapidly changing 
landscape, with some referring to an ‘audiovisual turn’ across a range of fields, 
from applied linguistics to media studies, and from musicology to philosophy. In 
the digital age, sonic elements often carry a significant functional load of textual 
meaning that is felt viscerally in the body (Cope & Kalantzis, 2020). Humans and 
civilisations encounter transformed soundscapes — sonic or acoustic environments — 
layers of meanings in our daily lives that are a vital part of our multisensorial 
emplacement in the world (Schafer, 1993). Elements of our perceived sonic exis- 
tence, whether of sounds, music, speech, or silence, form an invisible presence 
that can be read as an “auditory epistemology of everyday life” (Bull, 2000, p. 73). 

There has always been a fundamental relationship between spoken language 
and formal literacy learning because reading and writing are developed on the 
basis of one’s oral or face-to-face language development. Being able to hear and 
produce the 44 speech sounds of English, as well as hearing the syntactic and 
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semantic patterns of words spoken in phrases and sentences, is vital in learning to 
talk, and later, in learning to read and write (Tompkins et al., 2014). We know 
that children with severe to profound hearing impairment are placed at a distinct 
disadvantage to their hearing peers, in part because of the discrepancy between 
their incomplete spoken language system, and the related demands of reading a 
speech-based language system. Conversely, when children who are deaf or hard- 
of-hearing receive cochlear implants early in their language development (below 
the age of three), their rate of language growth becomes similar to their hearing 
peers (Geers, 2006). 

Hearing and producing sounds have always played a vital role in the body— 
mind—world nexus of literacy practice. In the case of speech, embodied activity 
that includes gestures, facial expressions, posture, head and body movements, and 
eye contact help to establish the goals of a speaker that carry a considerable pro- 
portion of the speaker’s meaning, while at the same time facilitating the speaker’s 
spatial memory and lexical retrieval (Gibbs, 2005; Morsella & Krauss, 2004). 
When people read aloud, motor areas of the brain are activated, particularly 
when reading action words or verbs, since Broca’s area is activated when people 
simply think about movement (Pulvermiiller, 1999). 

Speech production and singing, while able to be rendered digitally and syn- 
thetically (Cope & Kalantzis, 2020), are embodied modes that involve multiple 
bodily systems of voice production, including the air pressure, and vibratory and 
resonating systems. Produced by the body, speech and singing are undeniably 
embodied forms of meaning making involving the materiality of the muscles, 
vocal anatomy, and physiology of the singer. Likewise, noise is perceived bodily 
through the physiology of sound perception. Speech and singing are not only 
produced and perceived unmediated through the body, but the technologies 
for disseminating sounds have, over many decades, continued to become more 
compact, mobile, and wearable. 

Sonic elements of digital texts, such as the qualities of recorded voice, are 
imbued with meanings that semiotic theorists have analysed using parametric 
systems. For example, van Leeuwen (2017) identifies key elements of sound 
quality as pitch range (high-low), loudness (loud-soft), articulation, and reso- 
nance, among other features of the voice-quality parametric system (e.g. rough/ 
smooth, breathy/non-breathy, vibrato/plain, nasal/non-nasal). Van Leeuwen 
does not see that there is a chasm between semiotic accounts of speech through 
parametric systems, and the elements of voice that are uniquely constituted by 
the depths of one’s physiology or materiality of the singing or speaking body. 
Rather, one’s voice “can ultimately only be understood on the basis of our bodily 
experiences” (van Leeuwen, 2017, p. 77). Likewise, the semiotics of vocalisations 
is understood “by paying close attention to the physicalities of articulation” (van 
Leeuwen, 2017, p. 77). 

The digital transformation of audio-visual media has been significant in the 
20th century, as technologies for recording, amplifying, and distributing sounds 
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(speech, music, sound effects, and silence) moved from cylinder phonographs and 
nitrate-based film to disc records (acetate and vinyl), compact cassette, 8-track 
tape, polyester film, floppy disc, compact disc, mp3, video games, and more 
recently, web video, streaming apps, virtual reality, and 360-degree film (Müller, 
2020). Devices for transmitting digitally mediated sound are a growing part of 
wearable technology trends, such as wireless Bluetooth earbuds, headphones, 
Bluetooth headbands, smartwatches, and mixed reality smart glasses, that soni- 
cally augment the body anywhere, anytime, overlaying simulated soundscapes 
that often resonate with one’s social and cultural identity. 

There are many examples of crossmodal embodied relationships between 
aspects of hearing and literacy. The effects of listening to music while word pro- 
cessing, for example, has been researched, showing that participants write faster 
and more accurately in conditions of silence than when listening to background 
music. As 45 college-level participants authored brief expository essays, the 
background music was found to place high demands on their working memory, 
disrupting their word-processed writing (Ransdell & Gilroy, 2001). 

Shared crossmodal matching tendencies have been found in terms of human 
sonic-gustatory patterns of meaning making. For example, when groups are 
asked to pair classical music pieces with fine wines, participants show signifi- 
cant agreement. Mozart’s Flute Quartet in D major was found to pair well with 
Pouilly Fume white wine. Participants were also found to enjoy the experience 
more and the wine tasted sweeter when listening to the matching classical music 
than when tasting the wine in silence (Spence et al., 2013). In other words, sonic 
experiences are not simply a unimodal activity of the hearing ear but are often 
perceived and interpreted synergistically with other embodied forms of knowing 
about the world. 


Smell: Olfaction in digital media 


Smell, or olfaction, is becoming increasingly relevant to digital media practices 
and communication, as technological advances in digitally mediated olfactory 
displays for the simulation of smell are developed for virtual reality environments 
and other digital communication applications, such as smell-o-vision, smell- 
enabled games, and scent technologies for other film and media (Olofsson et al., 
2017). Of the senses, many consider olfaction to be one of the more enigmatic, 
since perceptions of smell have been found to be influenced by culture and age 
(Murray et al., 2016), gender (Shih & Blignaut, 2011), in addition to individual 
life experiences and even mood (Ghinea & Ademoye, 2011). 

Smell is an important sense or perceptual function connected to memories 
and emotions, which influences the sense of realism, quality, and one’s affec- 
tive evaluation of experiences in multimedia contexts (Murray et al., 2016). 
Olfaction has been found to have greater independence from other modali- 
ties (Danthiir et al., 2001). Until recently, olfaction has been one of the less 
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developed senses in the design of immersive virtual platforms (Howell et al., 
2016), and relatedly, less understood than the visual mode in literacy research 
(Mills & Dooley, 2019). 

A central aspect of literacy learning and reading texts is how words and their 
meanings are learned and processed in the brain. Olfactory research has dem- 
onstrated how the mind’s processing of words strongly associated with odours 
activates the olfactory regions of the brain in an embodied way. In an experi- 
ment with 24 subjects, simply reading words with strong olfactory associations, 
such as ‘fetid’, ‘cinnamon’, and ‘garlic’, without being exposed to any of the 
odours, immediately and automatically activated semantic networks in the olfac- 
tory cortices of the participants (Gonzalez et al., 2006). Smell is rarely referred 
to in teaching children to read, yet research points to the embodied nature of the 
development of our semantic networks associated with smell that are retrieved 
and activated during reading content with olfactory meanings (see Chapter 2 for 
other insights on olfaction and reading). 

Research has shown the connection between olfactory and auditory cues to 
support creative writing. For example, students were asked to write a short story 
in 15 minutes about an imaginary path on an island under four conditions — neutral, 
pleasant smell (e.g. coffee, laurel), sound (e.g. music), and smell with sound. 
Based on a number of qualitative and quantitative measures, writers were found 
to be more creative and expressive in the “smell” condition. Participants reported 
that the pleasant smell condition enabled them to feel “more relaxed”, “without 
any pressure”, “more immersed in the activity”, and allowed “a better flow of 
writing” (Goncalves et al., 2017, n.p.). 

Human communication and perceptions of others and our environments are 
influenced by the full sensorium and grasped holistically, with olfaction per- 
forming a critical role in communication, eliciting and processing emotions, 
retrieving odour-evoked memories, and supporting associative learning. Odours 
in human social contexts activate the amygdala-hippocampal complex for the 
emotional memory in an embodied way (Arshamuian et al., 2013). Adequately 
harnessing the power of scent-based technologies for communication, emotions, 
memory, and learning is now a key area in the development of immersive and 
digital learning games (Olofsson et al., 2017). 

Surprisingly, scented filmic media and gaming experiments were begun 
over a century ago, such as the theatrical use of rose essence which augmented 
a newsreel screening on the Pasadena Tournament of Roses Parade in 1906 
(Olofsson et al., 2017; Paterson, 2006). Today, 4D film or cinematic experiences 
digitally and mechanically stimulate five senses using wind, water spray, scents, 
lightning, flashes, fog, motion simulators in chairs, and with back and leg ticklers 
to engage the viewer using multiple senses. Off-the-shelf devices for olfaction in 
contemporary multisensory media are becoming more readily available, such as 
a growing range of scent diffusers and electrical interface output devices (Saleme 
et al., 2018). 
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Smell training through digital odour learning games has demonstrated the 
potential for enhancing cognitive function (Olofsson et al., 2017). Researchers 
look towards future taxonomies for smell for a wide range of communication 
purposes, since olfactory taxonomies have often prioritised research for certain 
industries and markets, such as wine, perfume, and food (Murray et al., 2016). 
Future smell-enabled media will be used for sharing and communicating with 
friends, supporting online purchasing decisions, mood and stress regulation, and 
scent-enabled immersive virtual media (Obrist et al., 2014). Others have begun 
to develop serious games and brain training applications to enhance our human 
olfactory capabilities as an important end in itself, particularly given olfactory 
association with enhanced cognition and memory (Olofsson et al., 2017). 


Taste: Gustation in digital media 


Like olfaction, taste — or gustation — has been one of the marginalised human 
senses in studies of new literacies, multimodal literacies, and communication 
studies (Mills, 2016) compared to visual, audio, and haptic media in human- 
computer interaction (Ranasinghe & Do, 2016). In fact, taste is currently 
regarded as one of the final frontiers in trajectories of simulated or immersive 
digital media technologies like virtual and augmented reality (Ranasinghe et al., 
2012). Gustation is an important sense associated with cultural practices that 
involve connecting with others over food and beverages, while different taste 
sensations are often connected to personal memories, emotions, and everyday 
interactions with the world (Ranasinghe et al., 2011). 

Research has demonstrated that gustation and literacy are connected through 
the body. For example, we know that gustatory disgust or a ‘bad taste in the 
mouth’ influences moral judgement when presented with a story or moral 
vignette. One experiment found that after participants consumed various 
beverages — sweet, bitter, and neutral (water) — the taste of the substances signifi- 
cantly affected moral judgement. Behaviours described in the moral vignettes 
from Wheatley and Haidt (2005) included stories of a congressman accepting 
bribes, a man eating his already-dead dog, shoplifting, and a student stealing 
library books. Participants used a multipoint judgement scale from ‘not at all 
morally wrong’ to ‘extremely morally wrong’. Gustatory disgust in the bitter 
condition elicited greater moral judgement of characters in the vignettes than 
in the sweet and neutral taste condition (Eskine et al., 2011). This supports the 
recognition that gustation and the moral judgements of characters that we read 
in stories are part of a tightly coupled embodied system. 

In another example of the crossmodal correspondence between taste (gus- 
tation) and literacy, researchers discovered that even the typefaces that we use 
in word processing are associated in consistent and predictable ways with the 
major gustatory categories of sweet, sour, salty, and bitter, and this is based on 
the roundness and angularity of the typeface. Participants associated angular 
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typefaces with bitter, sour, and salty tastes, while rounder typefaces were associ- 
ated with sweetness. Interestingly, rounder typefaces were also judged as more 
likeable and easier to read than angular typefaces (Velasco et al., 2015). 

In terms of recent digital developments, two commonly known gustatory 
technologies for multisensory media are lollipop and beverage output devices 
(Saleme et al., 2018). For example, researchers have developed the ‘digital lol- 
lipop’ for simulating gustation by varying the frequency, polarity, and magnitude 
of electrical currents that are applied to the tongue (Ranasinghe & Do, 2016). 
The simulation of taste for four major flavour categories — sweet, sour, bitter, 
and salty — are produced using different currents on the relevant regions of the 
tongue. The sweet taste emerges via an inverse-current mechanism, while the 
sour taste simulations were controlled to produce three sour intensities — mild, 
medium, and strong. Such media may be used to enhance literacy experiences, 
particularly for digital gaming, social media, and filmic media. 

Another novel example of utensil and beverage-based gustatory simulation is 
the platform Taste+, which involves two devices — a spoon and a drink bottle — 
that have embedded electrical modules to simulate three states of enhanced taste — 
sour, salty, and bitter. The Taste+ bottle uses electrodes on the tongue through 
a specially designed mouthpiece, supported by LED coloured lights, to alter the 
colour of water and has a mechanical dial used to select the flavour. The digital 
spoon enhances the saltiness and sourness of what is eaten from the spoon using 
two buttons on the handle, and electrodes on the mouthpiece (Ranasinghe 
et al., 2014). 

There are emerging possibilities for simulated communication environments 
that harness the meaning-making potentials of taste simulations, from virtual and 
augmented reality games, 360-degree film and 4D cinema, gustatory enhanced 
online interaction, and experiences involving eating and drinking beverages 
with others on social media who are geographically remote. For example, dur- 
ing the COVID-19 pandemic, special occasions such as birthdays have been 
celebrated by virtual meetings, where shared gustatory experiences could be 
enhanced. Recent technologies have been developed that allow digital users 
to share tastes remotely through devices that are used in conjunction with the 
internet (Ranasinghe et al., 2011). Future multisensory media environments will 
continue to develop new ways of capturing, rendering, distributing, and syn- 
chronising multiple sensory effects (Saleme et al., 2018). 


Materiality in literacies and digital media practices and 
embodied cognition 


The plethora of new digital technologies that have arisen in recent decades have 
a widening array of material forms that involve communicating and making 
meaning in tangibly diverse ways that matter for the mind. Information — depicted 
through words, images, audio, vibrations, and other modes — is recorded, 
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displayed, transmitted, and reproduced on computer screens, mobile devices, tab- 
lets, e-book readers, wearables, drones, robots, and other hybrid digital artefacts. 

This changing materiality of literacy practices is not an exclusively recent 
phenomenon — reading and writing have always made use of the cultural and 
material resources at hand. For example, early symbol-making surfaces or sub- 
strates by Indigenous peoples include cave walls and human skin in ceremonial 
body painting, as well as carvings with symbolic meanings on message sticks and 
musical instruments (Mills & Dreamson, 2015). The earliest known forms of clay 
tablet writing originated in Mesopotamia, where a stylus, thought to be made 
of reeds, was used to make wedge-shaped marks called cuneiform. Likewise, the 
ancient Chinese used carved inscriptions, as well as brushes and ink, on animal 
bones, called oracle bones, that were used in divination (Clayton, 2019). The 
materiality of encoding and decoding, and its relations with diverse bodily inter- 
actions, have changed across times and societies. 

Understanding the relationship between the embodied cognitive, cultural, 
and changing material technologies of reading and writing is a continually evolv- 
ing space in digital media research. In the context of the changing materiality of 
digital media practices the emphasis on embodiment counters views of literacy 
learning that focus on the mind or cognition alone (Haas, 2013). One of the key 
principles of embodied cognition is that humans offload cognitive work onto 
material environments as a cognitive strategy. Rather than manipulate exten- 
sive sets of symbols in our minds, we engage in forms of symbolic off-loading, 
essentially “leaving information out there in the world to be accessed as needed” 
(Wilson, 2002, p. 628). 

To manage the limits of our memory, our mental problem-solving abilities, and 
our ability to manipulate objects spatially in the mind, humans physically inscribe 
or manipulate symbols in the environment in the form of material notes, mud 
maps, calendars, back-of-the-envelope calculations, and many other forms of sym- 
bolic off-loading. In this way, encoding materials and technologies becomes part 
of our cognitive apparatus to support short-term memory and language function. 
Likewise, materials can become part of our long-term archiving of information 
that we cannot store with the same accuracy in the mind, or beyond our lifespan, as 
tangible technologies in our environment are used in archival ways, such as books, 
documents, digital files, and reference materials (Wilson, 2002). 

While there have been notable material shifts in literacy practices, from page 
to screen, desktop computer to tablets, mobile devices to wearables, what is sel- 
dom noted is the representational output bottleneck when writing and design- 
ing with computers. In other words, there is a limited range of material output 
devices by home users, with the predominant linguistic and graphic formats for 
multimodal designs comprising digital computer screens and colour printing 
(Seymour, 2011). The growth, affordability, and accessibility of additive manu- 
facturing, rapid-prototyping, laser cutting, 3D printing, screen printing, computer- 
controlled routing, and other forms of digital fabrication in recent decades has 
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opened an array of hybrid textual formats with a transformed materiality for 
multimodal representation and making. No doubt, the future of technology will 
see increasing capabilities for the digital and embodied expression of form-making 
across evermore hybrid material ways, gradually opening up literacy practices 
from comparatively narrow kinds of paper-based, screen-based, audio-visual, and 
document-based material formats. 


Tensions for engaging the full sensorium in digital 
media composition 


Despite the support for embodied literacy and digital learning activity in research, 
formal schooling worldwide has tended to work against ideals of embodied cog- 
nition by requiring students to sit for extended periods at desks, while emphasis- 
ing the performative aspects of ‘listening’ by controlling the expressive impulses, 
movements, and posture of the body (Luke, 1992). A criticism of research of 
embodied cognition in education is that often research focuses on the embodied 
action of the individual, rather than on the classroom collective, while research 
is yet to more fully account for how embodied cognition is shaped by the social 
context and groups within which learning occurs. In this way, research of 
embodied cognition and literacies can be aligned with socio-cultural theories 
of literacy learning, with individual cognition understood as connected to the 
broader social spaces in which digital media practices develop. A further tension 
is that there is often a lack of guidance for teachers to design literacies and digital 
media learning environments that optimise embodied learning in ways that have 
strong and meaningful task integration of sensorimotor action for optimal cogni- 
tion (Danish et al., 2020). 

One of the greatest constraints for introducing digital embodiment in mean- 
ingful ways in schooling is the ongoing digital divide. Research has consistently 
shown that access to technology alone is not sufficient to bridge the gap between 
the disadvantaged and the economically privileged (Warschauer & Tate, 2018). 
Even when technologies are provided to the marginalised, too often, the lack 
of infrastructure, commensurate training of teachers, development of curricu- 
lum, and related gains in educational outcomes are not given adequate attention, 
exemplified in the Los Angeles Unified School District iPad roll out (Blume, 
2015) and the Alabama One Laptop per child effort (Warschauer et al., 2011). 
During the pandemic in Australia, students from low-income backgrounds, 
rural, Indigenous, and differently abled students were the worst affected edu- 
cationally, particularly with lack of access to technology and reliable internet 
connectivity (Flack et al., 2020). 

Teachers in low-income schools have significant challenges in promoting digi- 
tal media inclusion, while research has shown that more technologies do not nec- 
essarily promote better literacy learning outcomes (Warschauer & Tate, 2018). 
Significantly, a wide variety of sensory literacy practices have been used in schools 
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that promote meaningful bodily engagement with few, old, or common tech- 
nologies, such as tapping out sounds, air writing, writing with fingers in sand, 
paint, or shaving cream, manipulating magnetic letters, and listening to audio- 
books while reading a printed book. Mobile phones are the most accessible tech- 
nology devices worldwide, including in developing countries such as Tanzania 
(Kafyulilo, 2014), while mobile phone-supported technologies for learning, such 
as e-books and augmented reality apps, are currently more widely accessible than 
more expensive immersive technologies. 

Some of the most significant challenges for implementing literacy curricula 
that optimise a broadened use of the senses are fundamentally ideological and 
historical because compulsory schooling in Western societies has often given 
priority to scientific rationalism, hierarchical views of the senses (i.e. vision at 
the top), and a strong emphasis on governance and surveillance. The measure- 
ment of abstract cognitive skills that are most easily tested and compared en masse, 
such as through bureaucratic and top-down standardised testing regimen and 
regulatory uses of big data, is used as arsenal in nation-state apparatus of control. 
Such practices too often militate against embodied forms of cognition, com- 
munication, cognition, and creative literacy practices that are situated in real or 
lifelike body—mind—world interactions that are often difficult to measure through 
conventional testing. 


Classroom implications of the sensory turn for literacies 


Literacies and digital media practices in social life will always be influenced by 
available technologies for reading, writing, listening, and viewing, with each 
digital media practice involving the senses and embodied cognition in particular 
ways. The body is constantly entangled as a vital sensorial instrument in a shift- 
ing print and digital media world, from walking or pacing to stimulate more 
creative oral language responses (Oppezzo & Schwartz, 2014), to smelling scent- 
based technologies for communication, emotions, memory, and learning in 4D 
filmic media (Olofsson et al., 2017). Bodily engagement matters whether listen- 
ing to music while enhancing one’s spatial-temporal reasoning (Hetland, 2000), 
or experiencing taste sensations that bring back personal memories and positive 
emotions while playing a virtual reality game (Ranasinghe et al., 2011). 

The implications for the sensory dimensions of literacy learning in the class- 
room are far reaching, with teachers having the opportunity to design learning 
environments and curricula to optimise a range of sensory literacy practices. As 
mentioned earlier in this chapter, it is not that more bodily engagement is neces- 
sarily better for literacy learning, but rather, that the body is guided to move in 
ways that are meaningful and relevant to the learning task at hand (Skulmowski & 
Rey, 2018). For example, using gestures to act out unfamiliar word meanings con- 
solidates vocabulary learning, while moving around in random ways that are unre- 
lated to the word meanings does not have the same benefit (Mavilidi et al., 2015). 
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Likewise, teachers have long sought to provide learners with opportunities to 
engage with texts in materially diverse ways, from handwriting and drawing on 
paper with pencils and crayons, to keyboarding and using tablet touchscreens. 
Educators need to be equipped with knowledge of how the material affor- 
dances of different literacy technologies, such as varied writing substrates (e.g. 
paper, canvases, smartboards, touchscreens) and writing implements (e.g. pencils, 
pens, styluses, 3D pens, keyboards), support different outcomes for embodied 
cognition. 

For example, graphonomic research has demonstrated that the fine-motor 
formation of letters using conventional handwriting implements, such as pencils, 
plays a vital role in the development of young learners’ letter perception abilities 
and letter categorisation skills, with children developing more enhanced word 
identification skills after writing words (visual-haptic) than after viewing them 
alone (Mangen & Balsvik, 2016). Likewise, tracing letters using fingers is shown 
to improve letter knowledge and decoding (Bara & Gentaz, 2011). Sensorimotor 
aspects of early writing are shown to be important for both short-term and long- 
term language learning, and the losses of not handwriting are exhibited in adults’ 
declining abilities to draw straight lines (Mangen & Balsvik, 2016). Keyboarding 
does not facilitate the same cognitive outcomes for early letter recognition but 
can help students to recognise variable font styles, while preparing students for 
computer-based writing in study, life, and work (Li & James, 2016). 

Some of the key growth areas of embodied technologies for meaning mak- 
ing in education are e-learning, digital games, video-assisted learning, mobile 
devices and tablets, wearable technologies, and immersive augmented and vir- 
tual reality. In higher education, courses such as Massive Open Online Courses 
(MOOC) may use blockchain technology, while artificial intelligence, such as 
AI tutors, chatbots, and automated grading will support learning and marking 
writing. Big data and learning analytics will continue to be more widely used for 
monitoring and supporting learning, while social media continues to support the 
dissemination of videos and resources among learning communities (Bui, 2020). 
Each of these trends has specific implications for embodied cognition and mean- 
ing making, with digital tools changing the way texts are created, read, viewed, 
assessed, and shared by learners. 


Future directions for researching embodiment in digital 
media practices 


In the future world of work requirements for literacies and qualifications for 
Industry 4.0 — the next industrial revolution — will influence new directions of 
research of digital media practices and embodiment. Conceptions of literacy 
and technology practices have been steadily influenced by successive industrial 
revolutions — mechanisation, electrical energy, and electronics and automation — 
along with consequential changes across all levels of the curriculum. Asa corollary 
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of these revolutions certain occupations are replaced, while new professions, 
formerly unconceived, have emerged (BeneSova & Tupa, 2017). Amidst these 
transformations, conventional embodied relations between literacy practices and 
their mediating technologies have been destabilised, with their modified physi- 
cal and virtual materiality — from robotics to additive manufacturing and 3D 
printing applications — involving multimodal modelling of data in fabrication 
industries. Rapid prototyping technologies, 3D hubs, fab-labs, and makerspaces 
will increasingly be used in earlier levels of schooling to induct students into life 
and work in the new digital age (Ford & Minshall, 2019; Kostakis et al., 2015). 

Digital communication environments and globalisation are two key drivers 
of educational change requiring hybrid ways of teaching and doing literacy as 
we have known it in the past. While much reading and writing has shifted 
from paper to screen, rapidly changing online communication environments 
now require multimodal text composition (e.g. digital image, audio, video, 3D 
models), while digital comprehension involves complex ways of synthesising vast 
amounts of information across multiple media formats via the internet (Leu et al., 
2015). Each of these changes to literacy practices influences embodied cognition 
in diverse ways that, to date, have not been fully explored. 

Further research will be needed to examine the relationship between 
sensorimotor elements of new digital reading and composition practices 
for embodied cognition and communication. Internet-based and hypertex- 
tual reading involves strategies that have no counterpart in traditional offline 
reading, requiring navigating reading paths in a shifting visual and problem 
space (Leu et al., 2015; See Chapter 2). Additionally, participation in digi- 
tal communication environments requires the development of new trans- 
cultural digital literacies and digital citizenship skills (Rapanta et al., 2021; 
Third & Collin, 2016). Students need to negotiate transnational connections 
with others, representing their identities in hybrid material and virtual ways 
for multicultural audiences via the web, accessed through computers, mobile 
devices, and wearables (Kim, 2016). 

The future of the sensory turn in literacy practices remains relatively 
uncharted territory for researchers and educators, as emergent media shape the 
reading and writing landscape. A new generation of literacy and digital media 
scholars are embracing the realisation that the body matters to literacy practice 
in essential ways that are fundamental to the mind and to meaning making. The 
implications for schooling are far reaching, as the entire educational enterprise is 
now grappling with the rise of global pandemics and natural disasters. 

Consequently, educational systems and teachers are delivering remote and 
online schooling impromptu and en masse, which has propelled thinking and 
scholarship globally towards new ways of doing school, of schools without walls, 
and of schooling anywhere, anytime. By necessity, literacies continue to be intri- 
cately connected to communities of practice that are strongly intertwined with 
home life, and to the wider digitally social world in which we live. 
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HAPTICS AND MOTION IN LITERACY 
PRACTICES WITH DIGITAL MEDIA 


Why touch and motion matter to literacy practices with new 
digital media 


Multiple sensory modalities are central to literacy practices, old and new, digi- 
tal, and non-digital, used to both perceive and express information, including 
sign making and creative textual production (Mills, 2016). From ancient cave 
wall inscriptions to clay tablets, and from papyrus to birch bark, wax, cloth, and 
parchment, writing and representation has always involved the materiality of a 
medium, in addition to haptic and cognitive skills (Bolter, 2001; Haas & McGrath, 
2018). Haptics here refers to sensory tactility or touch (Paterson, 2007). Today 
human beings interact with multimedia textual environments using a vast array 
of movements of the hands, supported by other larger limb and body move- 
ments, such as in the case of documentary filmmaking on location (Mills et al., 
2013). The current expansion of technologies and the changing materiality of 
media for interacting with texts has opened an ever more complex array of lan- 
guage practices involving touch and motion in new ways — embodied ways that 
are also intrinsically connected to cognition (Gibbs, 2005). This chapter explores 
the role of touch and motion in a literacy landscape that has, until recently, given 
greater priority to the visual mode in textual practices, including in theories of 
multimodality and new digital media (Minogue & Jones, 2006). 

Theorists have long identified a Western cultural bias towards the privileging 
of vision and the consequential backgrounding of other senses, such as touch 
and motion. A hegemony of vision has pervaded contemporary culture, while 
also framing understandings of learning, education, and literacy practices. This 
ocularcentrism has been associated with empiricist views of knowledge, where 
what is perceived and observed through the eyes counts as truth in science and 
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other knowledge domains (Mills et al., 2018; Pallasmaa, 2005). When theorising 
literacy practices, we see that research has typically focused on either the cogni- 
tive or social dimensions, while the role of the body in literacy practices has only 
recently received serious attention (e.g. Bezemer & Kress, 2014; Jewitt et al., 
2021; Walsh & Simpson, 2014). Mangen & Velay (2010, par 54) have made this 
argument quite strongly: 


Currently dominant paradigms in (new) literacy studies (e.g., sociocultural 
theory) commonly fail to acknowledge the crucial ways in which different 
technologies and material interfaces afford, require, and structure sensori- 
motor processes, and how these in turn...shape, cognition. 


While it might seem that touch and movement are supportive or peripheral to 
the main work of literacy practice, researchers have demonstrated that haptic 
perception and movement are indeed important because readers and writers 
are always in touch with the text that they interact with — cognitively, phenom- 
enologically, materially, and physically (Haas & McGrath, 2018). For example, 
Mangen and Velay (2010) discovered that the sensorimotor interactions of writ- 
ers with physical writing tools influence cognitive processing in significant ways. 
Various regions of the brain are activated when one types letters using a key- 
board, compared to handwriting a text. Visual attention was found to be more 
focused when writing by hand, with space and time converging, hand and vision 
conjoined. In contrast, keyboarding involves the splitting of the visual space and 
the haptic or motor space (Haas & McGrath, 2018). Another study found that 
students perform higher on conceptual application of lectures when they take 
notes by hand, compared to notetaking with a keyboard, a condition in which 
students tended to record more verbatim language and a greater number of words 
at the expense of deeply processing the knowledge (Mueller & Oppenheimer, 
2014). Clearly, haptics or touch is vitally connected to literacy practice and the 
workings of the mind, rather than detached or incidental to it. 

Haptics is superior to vision at perceiving certain properties of the physical 
world, including microspatial properties of viscosity, elasticity, compliance of 
materials, and pattern. Likewise, touch is the human sense most able to differ- 
entiate between textures, like rough and smooth, hard, and soft, wet, and dry, 
sticky, and slippery (Minogue & Jones, 2006; Zangaladze et al., 1999). Touch 
can be used to perceive the temperature of objects (cool, warm, hot), and can be 
used to discern the weight and volume of objects held in the hand (Minogue & 
Jones, 2006). Contrastingly, human vision is capable of discerning colour and 
macro-geometric properties, such as shape. At the same time, haptic and visual 
information are often processed in complementary ways, with vision discern- 
ing macro-geometric properties, and touch perceiving microspatial attributes 
(Verry, 1998). 
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When we consider the vast array of haptic and movement-based interactions 
that humans can experience through digital media practices, the possibilities and 
complexities begin to be seen. For example, Friend and Mills (2021) generated 
a typology of touch derived from the analysis of children’s touch in an interna- 
tional array of education settings and maker spaces. Students were found to use 
explorative touch deliberately enacted to explore the world, technologies, materi- 
als, and texts. Students often use auxiliary touch, which is touch via tools such as 
through a paintbrush, pencil, or keyboard. Evocative touch is tactility that evokes 
feelings, memories, or connections, such as when one runs their fingers over a 
finished sculpture emotively. Finally, students drew upon creative touch, touch that 
is inspired by the mind’s ingenuity, such as when in a state of flow while draw- 
ing a sketch (Friend & Mills, 2021). Literacy practices entail interpreting and 
producing representations of knowledge that involve sensorimotor origins and 
embodied cognition. 

Motion or body movement is similarly replete with different perceptual 
and meaning-making potentials from those of touch and vision, while also 
functioning in complementary ways to the other senses. For example, recent 
research shows that humans are better at visual motor targeting tasks while 
walking (e.g. press if it is a red target, but do not press if it is green). Pointing 
accuracy improves when in a state of locomotion, as opposed to when stand- 
ing still. This suggests that walking facilitates the flow of visual information 
and action from the environment in a perception-action loop, supporting 
cognitive activity and decision making in rapidly changing environments 
(Mokhtarzadeh et al., 2021). Conversely, multisensory research from the cog- 
nitive sciences has shown that restricting the use of the senses to vision results 
in perceptual processing deficits, as well as reduced efficiency for memory, 
learning, and communication, which includes writing and speaking (Shams & 
Kim, 2012). 

Theorists, such as Ingold, have long argued that locomotion is a portal for 
accessing memories and understanding place and “culture on the ground” 
(Ingold, 2004, p. 166). Literacy research has demonstrated the role of locomo- 
tion across a range of meaning-making situations, such as when scaling walls 
or moving with the camera in documentary filmmaking (Mills et al., 2013). 
Locomotion is central to virtual reality media designing of three-dimensional 
representations with motion sensors (Mills & Brown, 2021). Similarly, move- 
ments of the feet are an important part of Australian Indigenous ‘Welcome to 
Country’ ceremonial practices, where body movements and smoke are used to 
sensorially and symbolically clear away the wrongdoings of the past (Mills & 
Dooley, 2019). The changing potentials of haptics and motion in digitally medi- 
ated practices — from using a 3D pen to using wearable technologies while on the 
go — add a new layer of complexity to understanding and theorising the material 
and virtual enactment of literacy practices. 


110 Body and senses 


New opportunities for touch and motion in literacy practices 
with digital media 


In contemporary digital society, most of our reading and writing is done using 
a digital device, whether on a computer, mobile phone, tablet, e-book reader, 
or other digital display. Many people also use pen and paper to make notes, sign 
documents, personalise greeting cards, or to keep a private journal. A more 
personal form of communication that can reveal traits of the writer (e.g. shaky 
writing of the nervous), handwriting is a unimanual activity, while keyboard- 
ing on a computer or other digital device involves distinctly different bimanual 
haptics. For example, in handwriting, the writer forms each letter using grapho- 
motor skills to represent conventional shapes of letters. Contrastingly, there are 
no graphomotor skills involved when using a QWERTY keyboard, since the 
writer's task is to locate and press the letters on the keyboard (Mangen & Velay, 
2010). Since the advent of word processing software, computer-mediated writ- 
ing of any kind radically alters the haptic dimensions of the writing process and 
continues to do so as technologies evolve. At other times, haptics may not be 
involved much at all, such as when using voice dictation for text messaging with 
a virtual assistant. 

Communicating with others through oral language, the foundation of lit- 
eracy learning involves observable movements of the body, including motor 
control of the tongue, lips, and head, and supporting movements and posi- 
tioning of the eyes, head, torso, arms, and legs (Gibbs, 2005). Similarly, 
reading, writing, and other literacy practices are not exclusively performed 
while seated in a single location but are often carried out while on the 
move. Many digital media and multimodal language practices are supported 
by movement of the whole body, including the action of the legs and feet. 
For example, the movement of the body is vital across a range of expressive 
multimodal practices, from non-digital language practices, like spoken pre- 
sentations, role plays, discussions, and interviews, to digital practices such 
as filmmaking, interacting with information in virtual reality simulations, 
programming an animated story, or creating a 3D model using a mixed real- 
ity headset (e.g. smartglasses). 

Digital reading and information practices that depend on movement include 
interacting with a 360-degree film, using augmented reality apps with geo-spatial 
tracking, navigating a journey using Google maps, using an activity tracker, read- 
ing an e-book on public transport, reading digital signs and environmental print, 
using a self-serve checkout, and many other social situations. A clear example 
from classroom research involved students choreographing a dance and program- 
ming a complementary virtual character’s dance based on curricular content, 
which involved an assemblage of multimodal literacy practices to improve stu- 
dents’ computational thinking (Leonard et al., 2015). Interestingly, the students 
first created the dances with their own bodies, which enabled them to easily 
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transfer the patterns of movement when programming the robots, demonstrating 
the centrality of movement in a digital media practice. 


Bodily basis of meaning making and early language learning 


One might argue that in many of these movement-based literacy situations, 
the body’s location and movement is merely peripheral or backgrounded, 
while the workings of the mind at an abstract level is dominant and somewhat 
removed from immediate perception. However, a growing body of evidence 
has indicated that all cognition is body-based (Mills & Exley, 2022; Wilson, 
2002). Research on embodied cognition draws attention to mind-body pro- 
cesses, which counters the Cartesian dualistic view that separates perceptual 
experience from cognition or reasoning (Mangen & Velay, 2010). Many key 
features of cognition are, in fact, dependent on perception, movement, and 
action, with the body playing a fundamental role, often working subconsciously. 
For example, gestural movements are not only used for speaking and commu- 
nicating information but play an active role in supporting fluency of thinking 
(Pouw et al., 2014). Research has shown that human evaluative judgements 
based on presented information are more positive when the research partici- 
pant is simultaneously pushing upwards on a table (Robbins & Aydede, 2009). 
Likewise, abstract concepts are developed fundamentally from metaphors that 
are first realised as bodily experiences and interaction in real-world environ- 
ments (Cox et al., 2017). Counting is first performed using fingers or one-to- 
one correspondence between the hands and the objects, to be later rehearsed 
and recalled in the ‘mind’s eye’ as mental arithmetic. Likewise, early language 
learning is aided by music, clapping, and rhythmical actions, and body move- 
ments to discern patterns in language. When children learn to talk, one of the 
earliest necessary skills is for the infant to understand the meaning of the care- 
giver’s pointing gestures as an exophoric reference to the person or thing being 
named. The development of this shared intentionality is vital in vocabulary 
learning and social development. As Goldin-Meadow (2007, p. 741) has argued, 
“children enter language hands first”. 

When children learn abstract graphic forms, control groups show that those 
who both study the forms visually and trace the forms haptically with their index 
finger have better memorisation of the graphic items compared to those who 
only learned them through the visual mode (Hulme, 1979). In other words, the 
haptic and visual information is better combined for such tasks, such as letter rec- 
ognition, than reliance upon visual memory (Mangen & Velay, 2010). Similarly, 
researchers demonstrated that when pre-reading children and adults were learn- 
ing to recognise symbols and letters, characters that were learned by copying the 
letters using handwriting were recognised more accurately than symbols that 
were learned by typing them using a keyboard, which applied to both pre- 
reading children (Longcamp et al., 2005) and adults (Longcamp et al., 2006). 
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Haptics in digital media practices 


Researchers acknowledged in the 1950s that communication tends to be con- 
sidered as perceived through the eyes and ears and that other sensory channels 
are forgotten (Parisi & Archer, 2017). When we look at the history of the senses, 
there have been calls to acknowledge haptics or touch across a range of media 
practices for some time. Walter Benjamin in 1935 (2008) drew attention to the 
role of the hand in digital photography in that the finger arrests a view of reality 
at the press of the shutter release button, mediated through the photographer’s eye 
and position of the body (Parisi & Archer, 2017). Similarly, in the field of human— 
computer interfaces, research since the 1970s has aimed to develop new media 
that would centre touch the way television had centred vision, and radio had done 
for sound (Noll, 1971). Others in psychology, such as Sherrick (1975), observed 
the paucity of psychological research on touch compared to vision and hearing, 
calling for investigations in skin-based communication (Parisi & Archer, 2017). 

Many educators see the potential of video game play as interactive literacy 
practices because they call on the user’s knowledge of narratives, decision mak- 
ing, critical literacies, multimodal texts, and gaming paratexts (game fan sites, 
wikis, books, magazines, etc). Haptics has become more visible in the design 
of game controllers, which since the 1990s has moved towards more embod- 
ied user interfaces (Fishkin et al., 1998). Games began to involve more direct 
physical manipulation of the technology to control events in the game, such 
as through tilting devices, shaking, or rotating them, or by blowing air directly 
into the microphone. Game controllers today also include vibrations or rumbles 
(Paterson, 2007), which have been adapted for use on mobile phones, and as 
reminders for sedentary wearers of smartwatches, Fitbits, and other wearable 
technologies (e.g. 150 steps to win the hour). Vibrations are difficult to ignore 
compared to screen displays, particularly when one’s concentration is focused 
elsewhere. These haptic sensations are sometimes referred to as ‘taptics’ or 
vibrotactile sensations, simulating human tapping on the wrist (Lupton, 2017; 
Paterson, 2017). 

Immersive virtual reality technologies with head-mounted displays support an 
array of sign-making practices, with motion tracking to allow users to perform 
a wide variety of manipulable haptic functions in a 3D world. Current examples 
include users shaping pottery from diverse cultural traditions (e.g. Pottery VR), 
virtual, in-air brush painting (Google Tilt Brush), and spray-painting graffiti (Vive 
Spray) on simulated buildings, subways, and walls. In some 360-degree interactive 
films, users can manipulate textual artefacts that are laden with print. For exam- 
ple, in the award-winning virtual reality film, The Book of Distance, Randall Okita 
engages in personal storytelling about his grandfather in Canada in the 1930s, 
who was imprisoned because he was Japanese. The virtual reality version of the 
film that can be played wearing a head-mounted display and motion sensors, 
invites users to touch, hold, and read archival family parchment, photographs, 
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passports, letters, and newspapers, and other texts in an emotionally moving, sim- 
ulated personal account of vibrant textual practices (Oppenheim & Okita, 2020). 
Because the user has a 360-degree view as a participant in the film, spotlighting 
is used to direct the reader’s gaze to interact with texts in turn. Once the user 
has finished touching and reading certain textual artefacts, the next part of the 
film unfolds, making haptics salient to the pacing and flow of the filmic narrative. 

Another innovative example of how haptics and the tactility of media are 
changing with new possibilities pertains to data materialisation. For example, 
researchers have used 3D printers to fabricate customised, edible chocolates that 
translate self-tracked heart rates after physical activity into a material form — the 
size of the chocolate reflecting the amount of physical activity undertaken by 
the participant as a reward. The chocolates were also inscribed with encourag- 
ing emoticons and words to support continued health gains. Personal digital 
data was no longer represented as abstract 2D mathematical and linguistic sym- 
bols, but could be viewed, touched, and tasted in novel, multisensorial ways 
that were memorable and meaningful for the participants (Khot et al., 2015; 
Lupton, 2017). 

Sign-making practices extend well beyond print, with designers and data art- 
ists creating materialisations of data through 3D fabrications of data sculptures 
that invite tactile interaction. Such practices involve the aesthetic physicalising of 
information. A good example is the weather bracelet by Whitelaw (2009) that 
translated meteorological data in Canberra to a tangible and familiar tactile form. 
The highs and lows of the jagged contours of the bracelet show the maximum 
and minimum temperatures for each of the 365 days of the year recorded by the 
Bureau of Meteorology for Canberra. Small holes show the weekly rainfalls. The 
bracelet makes weather data tangible, but also invites an intimate, tactile familiar- 
ity with local, place-based information that can be ‘read’ and felt through touch. 

Recent research on haptics in three different makerspaces with upper ele- 
mentary children explored how touch was orchestrated with other multisen- 
sory resources in students’ multimodal designs. Learners were engaged in making 
technology-enhanced modelling clay and recycled junk sculptures with moving 
parts, and sculptures with 3D pens. They made kinetic e-paintings combining 
conventional painting with embedded flashing lights coded using Arduino kits so 
that the paintings exhibited flow and movement. The students filmed ‘slime vid- 
eos’, a popular YouTube craze involving the tactile and acoustic exploration of 
the viscosity of slime. Students also engaged with non-digital forms of drawing, 
painting, and sculpting (Friend & Mills, 2021). The research highlighted distinct 
ways in which touch, as a central part of the sensorium, contributed to thinking, 
creativity, feelings, and multimodal representations of knowledge. Movement of 
the hands did not simply carry out the work of the mind, but touch was involved 
recursively. As one student explained: “I’m just drawing, and then I get ideas. 
I’m drawing a rainbow, but it will probably change”, while another said: “I’m 
using touch as in...if I can connect myself to the artwork, it just seems to flow 
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better” (p. 10). In other words, sometimes touch leads the direction of the mind, 
in creativity and designing, rather than simply following. 


Body motion, language learning, and digital media practices 


Human language and thought are fundamentally grounded in the felt experi- 
ences and action of the body; likewise, language and cognition are developed 
through the performance of a recurring bodily activity (Gibbs, 2005). Language 
and literacy practices can be understood by examining the ways in which the 
body engages with the tangible world — people, objects, texts, and other experi- 
ences in the environment. This recognition is supported by research in embod- 
ied cognition and language, which attends to sensorimotor experiences in 
cognition (Wellsby & Pexman, 2014). This view differs from classical cognitive 
science which gives priority to the minds’ processing of abstract symbols, while 
de-emphasising the role of the body. 

While theorists such as Piaget (2005) have drawn attention to the role of 
sensorimotor experience in childhood, with humans showing greater ability for 
abstraction later, recent theories of embodied cognition show that the role of 
embodiment in conceptual processing 1s always present, recurring, and continues 
through the life course (Wellsby & Pexman, 2014). In other words, the function 
of sensorimotor experiences does not stop or change fundamentally throughout 
development, but rather, becomes more agile, refined, and specialised over time 
as humans acquire new experiences and knowledge from the environment that 
build on earlier experiences (Antonucci & Alt, 2011). 

Motion is vital to speech in the sense that speech and gesture activate each 
other, beginning firstly in infant development with manual motion, which forms 
the basis for the entrainment of speech production through the coordination of 
hand and mouth in speech (Iverson & Thelen, 1999). In other words, infants 
develop their representational system and speech through perceptual and motor 
interactions with their environment, sensorimotor experiences forming a critical 
part of language and thought. 

Research demonstrates that sensorimotor knowledge forms the basis of lan- 
guage and higher-order thinking development among children of school age 
(Kontra et al., 2012) and throughout the life course (Thelen, 2007). For exam- 
ple, humans draw on sensorimotor information that they have gained through 
prior experiences in the world to comprehend language as they simulate vocab- 
ulary and sentence meanings (Glenberg & Kaschak, 2002). Similarly, humans 
have been found to draw on sensorimotor experiences when they describe 
and understand the orientation of objects, such as the relative position of items 
(Stanfield & Zwaan, 2001), and when they explain object affordances, such as 
the functions of tools (Myung et al., 2006; Wellsby & Pexman, 2014). 

As children understand and build up a vocabulary of nouns — the name of 
an object, place, or thing in the world — sensorimotor experiences influence 
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their categorisation of objects and their vocabulary learning. For example, Smith 
(2005) conducted experiments with two-year-olds who were given two dif- 
ferent experimental conditions — one in which participants could move and 
manipulate an object before being asked to name and distinguish the object 
from among a group of very similar objects, categorising and describing it using 
appropriate vocabulary; and the other group that could only observe the object 
which the experimenter moved for them. The participants who were encour- 
aged to explore and move the object around were able to provide descriptions 
that showed more accurate language and judgements of shape, function, and 
categorisation decisions based on the performed action than the group that was 
limited to using the visual mode (Wellsby & Pexman, 2014). 

Experiments with toddlers have also been performed to understand how 
body positioning and spatial location influence vocabulary learning of nouns. 
They found that children used the location of labelled objects to reidentify them, 
not just attributes of the objects themselves. Toddlers also performed differently 
in their functional knowledge of objects and their label based on whether the 
child was standing or sitting, suggesting that children’s body posture can play a 
role in linking objects to their noun label (Smith & Samuelson, 2010). 

Similar findings were identified in children’s learning of new verbs because 
motor areas in the brain were activated only for verbs that the participants had 
explored through their own sensorimotor action, and not for verbs learned by 
hearing the label and watching the experimenter do the action (James & Swain, 
2011). When learning new adjectives pertaining to tactile qualities of objects 
(e.g. spongy, spiny), two-year old participants who were taught using tactile 
gesture, such as the experimenter squeezing the spongy object performed better 
than those taught through the experimenter pointing to the object (O’Neill et al., 
2002; Wellsby & Pexman, 2014). 

In contemporary digital media environments, the use of anywhere, any- 
time literacies through mobile technologies — invoking locomotion and 
other emplaced and embodied meanings — is worthy of research attention. 
Technologies that are used while on the move are now ubiquitous, such as 
video cameras, mobile phones, tablets, and other wearable devices, inviting 
greater involvement of large-body movement, such as walking or locomotion. 
Mills et al. (2014) describe research in which children moved through differ- 
ent natural and built environments while walking with video cameras to cre- 
ate documentary films about healthy places and the connection to emotions. 
Locomotion and moving through places on foot was essential in this process, as 
children captured and reflected multimodally on film, the materiality of lived, 
embodied, and situated experience of places (Mills et al., 2013). A repeated 
motif in the films was that students often recorded their feet walking to loca- 
tions, along walks and footpaths, while lying on the ground to admire the blue 
skies, and in front of them as they glided down playground slides. The research 
highlights the way in which knowledge making and representation through 
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documentary filmmaking on location, directly involved active participation 
through practical bodily and sensorimotor exploration, locomotion, and lived 
experience of places (Mills, 2016). 

Others such as Ehret and Hollett (2013) have explored how middle-school 
adolescents produced texts and media through mobile devices (iPod touch) 
while wayfaring around school spaces — a distinct form of ambulatory and 
embodied movement that is not focused on simply getting to a destination — 
theorised by Ingold (2011). Researchers focused specifically on how students 
moved with their devices and the effect on their composition as they took pho- 
tographs, film, and wrote. They observed that students became more attuned to 
the features and affordances in the environment for composing, while producing 
texts that challenge those scripted for formal school-based literacy practices and 
routine uses of the spaces in the school. 

A growing range of digital media, from immersive virtual reality to mixed 
and augmented reality technologies, and from robotics to drones, have opened 
fundamentally different opportunities for understanding and optimising body 
motion and locomotion for literacy learning. Researchers and teachers who have 
access to novel media can teach students to encode and decode language using 
new virtual overlays, augmented reality books and apps, 3D holograms, simula- 
tions, 360-degree films, and other digital texts that have multisensory elements, 
and which track the users movement or geo-spatial positioning (see Chapter 7, 
this volume). The specific outcomes of these kinetic-based digital media prac- 
tices for multimodal composing and language learning, particularly their body- 
based potentials for extending human thought and language, remain largely 
uncharted territory. 


Tensions and challenges for touch and motion in literacy 
practices with digital media 


Drawing attention to the role of haptics and motion in literacy and digital media 
practices does not negate the realities of politicised notions of literacy standards, 
exams, and assessments, requisite for many higher education opportunities and 
employment, where linguistic competencies are given privileged status over 
non-linguistic modes (Pahl & Escott, 2016). Our argument is not that linguistic 
elements of literacy should not be given priority in the curriculum; rather, there 
are many haptic elements of writing, keyboarding, drawing, public speaking, and 
using new technologies for interacting with texts that should not be taken for 
granted, skills which often require many years of training to move from rudi- 
mentary levels to expert accomplishment (Haas & McGrath, 2018). 

Likewise, few have attended to the way in which literacy practices, thought, 
and language, are always first apprehended in movement, in forms of ambula- 
tory knowing and pedestrian activity of the limbs and body (Mills & Dooley, 
2019). The development of language and literacy practices are part of childhood 
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and maturation, as terrestrial and embodied beings, who without transporta- 
tion technologies, live and learn on the ground (Ingold, 2010). The repeated 
engagement in language and literacy practices, digital and non-digital, gradu- 
ally become part of one’s muscular consciousness, so that motion becomes 
taken for granted often until humans encounter an unfamiliar technology or 
tool when the haptic elements need to be learned with some intentionality. 
This has been seen, for example, when introducing a new virtual reality tech- 
nology to students, such as immersive, three-dimensional painting in a virtual 
world — a context in which students need sufficient time and guidance to 
become familiar with the haptic controls and affordances for designing (Mills & 
Brown, 2021). 

Yet many digital technologies for reading and writing are designed in ways 
that separate the workings of the hand or body from the visual space of the text, 
positioning the body in new ways, such as in the case of a computer monitor 
and keyboard (Haas & McGrath, 2018). Each new technology, from interact- 
ing with a 360-degree, virtual reality film, to filming movements to popular 
music excerpts on social media sites such as TikTok, calls for different kinds 
of body-knowing and body-communicating, with a vital interconnectedness 
between body, motion, space, and the mind (Ntelioglou, 2015). Perhaps some- 
what surprisingly, technology developers do not always seek to apply research 
from embodied cognition, phenomenology, and neuroscience about the con- 
nections between hand, body, and mind (Mangen & Velay, 2010). Allen et al. 
(2004, p. 229) argue: 


If new media are to support the development and use of our uniquely hu- 
man capabilities, we must acknowledge that the most widely distributed 
human asset is the ability to learn in everyday situations though a tight 
coupling of action and perception. 


When teachers implement learning experiences that draw on use of haptics and 
motion using digital media literacies in the curriculum, principles of good peda- 
gogy still matter, including the teacher’s discerning selection or adaptation of 
technology applications in which haptics and motion are meaningful and pur- 
posefully used. Likewise, any technology for literacy can be used for ideological 
purposes: no technology should be used uncritically (Haas & McGrath, 2018; 
Luke, 1992). The ethical and civic consequence associated with the use of any 
newly produced technology should always be critiqued, asking questions such as: 
Who benefits and who misses out? Whose political or commercial interests are 
served? and What digital data traces will be used and for what purpose? How will 
haptic activity and geo-spatial movements be traced and shared with technology 
developers, and for what purpose? Research has found that most users of digital 
technologies have little knowledge of how their personal data is used, where it 
goes, and in what ways third parties are involved in its use (Lupton, 2017). 
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Implications for digital media practices involving touch and 
motion in the classroom 


Research has long demonstrated the vital role of haptic play and motion from 
birth that equips humans to learn about the world through the use of the full 
sensorium (Heath, 2013). Brain development requires the stimulation of whole- 
body motion and haptic interaction, with hand-thought-language relations 
occurring during playful manipulation of the environment. This forms the foun- 
dation for early language and literacy learning, including elements of imaginary 
play that evokes story creation. Heath (2013, p.191) has elaborated on these early 
learning haptic events: 


The hand — as investigator and manipulator of the environment — calls on 
all the modal systems necessary to produce these representations, especially 
so when the hand is shaping, grasping, drawing...or moulding materials... 
Questions that ask, “What if? and ‘What’s this about?’ result from the force 
patterns the brain exerts through the fingers, hands, and forearms. 


Haptic feedback is vital in the performance of literacy practices, produced by the 
gripping action around media, such as crayons, pencils, and pens. Yet the body— 
brain—language connection for humans continues to develop throughout the life 
course, as the heuristic role of haptics and motion is developed through creative, 
skilful, and controlled use, such as for experienced musicians, orators, authors, 
dancers, artists, crafters, architects, graphic designers, stenographers, and many 
others. In terms of writing, for example, research demonstrates that handwriting 
speed and ease is not tangential to learning, but forecasts students’ future ability 
to compose lengthy texts (James & Engelhardt, 2012). At the same time, the rec- 
ognition that sensory learning should occur in the early years is not universally 
applied in classroom practice. Australian scholars have observed the reduction of 
multisensory learning environments across many early learning classrooms — a 
consequence of national assessment pressure for the development of formal lit- 
eracy skills to be taught fast and furious (Somerwil et al., 2020). 

A key implication of the essential role of haptics and movement in language 
and literacies development is that simply giving students opportunities to press 
buttons on digital game controllers or to swipe a touchscreen, does not guar- 
antee that language learning is occurring optimally (Heath, 2013). For instance, 
apps for early language learning often involve haptics and movement, but these 
are not all equal in terms of learning gains. For example, when Mills et al. (2018) 
compared The Heart and the Bottle in print and e-book versions, we found that 
only the e-book version uniquely invites the readers’ physical participation in 
activities of the main character that enables the reader to take part vicariously in 
the character’s experiences and feelings. An instance of this is when the reader 
must shake the mobile device or tablet as the character shakes the glass bottle to 
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release the visible heart inside, as well as sawing, drilling, and other actions that 
frustratingly fail to retrieve the heart. This kinaesthetic focalisation enables the 
reader to align their experiences and feelings with the character in meaning- 
ful ways (Mills et al., 2018). In contrast, interactive e-books can also be poorly 
designed, so that readers are distracted from the storyline with peripheral, gra- 
tuitous, or meaningless haptic actions, such as repeatedly touching background 
animations that are not relevant to the central narrative (see Chapter 10, this 
volume). 

It is important for students to experience a range of haptic and motion-based 
literacy activities that are both digital and non-digital. This is because research 
has shown that some dimension of touch-tablet experiences, for example, if used 
alone, do not supply many opportunities for students to apply varied pressure 
on objects to produce different effects, as occurs when increasing pressure to 
create thicker lines using pencils, crayons, and paint brushes (Crescenzi et al., 
2014; Neumann & Neumann, 2014). An exception would be the use of certain 
technologies, such as the pressure sensitive stylus pens and associated drawing 
applications, that simulate these effects of pressure to modulate line thickness. 

Tablets and virtual reality technologies also restrict certain rich, sensorial 
haptic experiences, such as being able to touch varied textures, determine the 
degree of wetness or dryness of surfaces, perceive the temperature of things, and 
feel the viscosity of flowing substances, such as paint, modelling clay, sand, and 
other representational media. At the same time, virtual simulations of creative 
activities, such as virtual painting and pottery making have different advantages 
for haptic and bodily movement that can be explored with older students, given 
that virtual reality technologies with head-mounted displays are currently mar- 
keted to older children and adults. 

A variety of digital and non-digital tactile and motion-based experiences that 
optimise meaning making should involve at least six distinct types of explor- 
atory touch (Lederman & Klatzky, 1987). These are lateral motion (to perceive 
texture), pressure (hardness), static contact (temperature), unsupported holding 
(weight), enclosure (global shape, volume), and contour following (global shape, 
exact shape). Additionally, digital controls and technologies can invite two other 
types of exploratory touch: ‘part motion testing’ which is the human ability 
through the hand to explore what moving parts can do, applying force to the 
part while stabilising the rest of the object. Likewise, ‘function testing’ involves 
executing haptic movements to perform certain functions, such as putting one’s 
hand inside a puppet, or testing the game control functions in a particular soft- 
ware application (Lederman & Klatzky, 1987). 

In terms of body movement and motion, literacy practices that afford oppor- 
tunities to use larger body movements contrast conventional reading and writing 
that is too often limited to fine-motor movements while seated or stationary 
indoors. Reading and composition can be digitally mediated by immersive 
virtual reality with head-mounted displays and motion tracking, mixed reality 
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headsets and augmented reality applications that track the user’s body, outdoor 
filmmaking and photography, and other activities involving encoding and decod- 
ing with mobile or wearable devices. 


Recommendations for researching touch and motion in new 
literacy practices 


As the world of media and communication technologies rapidly shifts around 
us, so too must research of literacy practices that engage new forms of haptics 
and motion. A large number of studies compare the haptic dimensions of older 
technologies, such as handwriting to keyboarding (e.g. Longcamp et al., 2005, 
2006; Mangen & Velay, 2010; Mueller & Oppenheimer, 2014) as well as a grow- 
ing number of studies involving touchscreen tablets for children’s learning (e.g. 
Crescenzi et al., 2014; Neumann & Neumann, 2014; Walsh & Simpson, 2014). 
However, there are new generations of digital applications, such as the Internet 
of Things (IoT) and wearable technologies (Lee, 2016), that have received lesser 
attention in terms of haptics and body movement, and particularly for literacy 
learning, representation, or decoding. 

There are also more studies of touch or haptics in literacy learning (e.g. 
Bezemer & Kress, 2014; Haas & McGrath, 2018; Heath, 2013; Hulme, 1979) 
than of body movement and locomotion. This is a significant gap in research 
that will need to be filled, particularly when examining literacy practices across 
cultures, such as Indigenous narrative practices and ceremonies that give prior- 
ity to painting of the body, dance, percussion, and movement — many of which 
have become part of a growing digital and social media heritage (e.g. Giaccardi, 
2012; Mills & Dooley, 2019; Mills & Dreamson, 2015). It is anticipated that 
the rapid growth and technology development in augmented and virtual reality 
technologies for educational purposes (e.g. Eldequaddem, 2019) will give rise to 
a new wave of research to identify constraints and affordances in terms of their 
materiality and their sensorial potentials for literacy learning. 
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VIRTUAL, AUGMENTED, AND MIXED 
REALITY 


New literate bodies 


Literacy and digital media practices, including multisensorial interactions, have 
undeniably changed in recent years, with transformed texts and the embodi- 
ment of practices through virtual, augmented, and mixed reality technologies. 
In societies in which there is a trend towards increasing digitalisation and detra- 
ditionalisation of conventional forms of communication, there are complexities 
concerning the very constitution of a text. 

One example within a VR environment is Let’s Create! Pottery VR where 
users can mould, bake, and paint life-sized, immersive, three-dimensional pot- 
tery with visual designs and hieroglyphics on a virtual wheel (see Figure 7.1). 
Such virtual texts are created while wearing specific apparatus: a headset to create 
an immersive textual experience — blocking out a view of the real world — and 
hand controls or sensors, like mechanical appendages that augment the user’s 
body. 

Or consider MR HMD smartglasses, which have a built-in microprocessor, 
such as Microsoft HoloLens. These hybrid technologies allow users to interact 
with floating touchscreens, control panels, and three-dimensional texts that are 
surreally anchored to one’s real or corporeal world, but which have no tangible 
substance. For example, Figure 7.2 shows screenshots of stories created by chil- 
dren who used the Hololens 2 MR headset to illustrate oral narratives using 3D 
holograms. This is one scene from each story, selected from a series of events. 
The first is a story about a unicorn finding treasure at the end of the rainbow in 
an arctic scene, while the second is about a superhero in a cityscape. The students 
used in-air haptics to manipulate the 3D models that were superimposed over 
the physical world. 

Such virtual, holographic texts, like manipulable sketches, labelled three- 
dimensional molecular structures, or forests of virtual trees superimposed on the 
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FIGURE 7.1 Roman pottery by an 11-year-old student using VR and in-air haptics 


FIGURE 7.2 Mixed reality narratives: 3D holograms viewed through the Hololens 2 


real world, are imperceptible to others who are not wearing the smartglasses. 
For those wearing the digital device, the holographic images can be viewed, 
touched, and directed using one’s fingers in the air. 

These augmented and virtual reality texts, with their new modal affordances 
for three-dimensional representation, will not eradicate literacy traditions and 
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routine bodily interactions with texts, such as using the alphabet, reading, 
spelling, drawing, or typing on a keyboard. Neither should these virtual and 
augmented texts be seen as diametrically opposed to, or competing with, con- 
ventional literacy practices, such as using newspapers, books, and pens on paper. 
Instead, their use can be complementary. For example, some researchers have 
begun to employ VR technologies to teach conventional literacy skills, such 
as English language learning or foreign language learning (Mills, 2022). AR 
technologies have been used to support early language acquisition reported in 
at least 52 studies, used for spelling games, word knowledge, interactive read- 
ing, and location-based word activities (Fan et al., 2020; He et al., 2014). What 
is emerging is a co-existence of multiple, differentiated, material, and virtual 
practices that are becoming more novel, affordable, and widespread, opening up 
an exponentially expanded number of representational choices, media, textual 
features, and hybridised practices for different social purposes. 

Such shifts call for the ongoing rethinking of what constitutes literacy and 
media practices. We need to consider: What existing skills will be relevant and 
what will become backgrounded or foregrounded, essential, or cursory in the 
identity performances today and in societies of the future? Who will be inducted 
into these hybrid assemblages of wearable media first, last, or never, and how 
often, and to what ends? What multisensory interactions, modes, bodies, and 
cultural content will be privileged in these virtual practices that are generated 
by technology developers in the entertainment and ‘edutainment’ industries? 
This chapter charts some of the new territory and implications of these recent 
developments that have taken the technology industry by storm (Johnson et al., 
2010), but for which the important technical facets that are common to these 
media, and that shape their potential as social practices, are largely unexplored. 

A range of virtual, augmented, and mixed reality contexts are opening up 
possibilities for literacy practice, collectively referred to as ‘extended reality’ 
(XR) that includes future technologies still in development (Mills, 2022). It 
is difficult to ignore that these platforms are often developed and used in the 
context of gaming and entertainment. While some applications are designed for 
productivity and commercial use rather than play, such virtual, augmented, and 
mixed reality platforms are typically supported by gaming wearables and hard- 
ware. Games are defined here, following Salen and Zimmerman (2004, p. 80) 
as a practice involving players who typically engage in artificial conflict within 
a system of particular rules with ‘quantifiable outcomes’. Games typically create 
an environment that contrasts with the seriousness of ordinary life, and in the 
context of education, with the seriousness of schooling, while often absorbing 
the player so that they become fully and seriously engrossed in the game. 

Researchers have argued that a key advantage of games for learning is the 
provision of space for experimentation, role play, and simulation that is less con- 
strained by conventional social norms. In many cases, such as for serious gamers, 
gaming is integral to the construction of social roles and identities (Garcia, 2018; 
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Scholes et al., 2021). VR, AR, and MR technologies have been identified as 
some of the fastest-growing developments in the commercial market, but also 
now in education, and predicted to increase exponentially in the next decade 
(Eldequaddem, 2019). 

So how do theorists define and describe these mediated technologies that 
have new literacy and social potentials? AR technologies overlay virtual content, 
such as animations, products, and other information, over the real world. In 
this way, elements of the virtual and real world coexist in useful ways for learn- 
ing, social purposes, and text creation (Chang et al., 2011). AR applications 
can employ a head-mounted or handheld display, webcam, or projector-camera 
system (Fan et al., 2020). The expansion of AR technologies has been strength- 
ened by the development of mobile devices with internet access, GPS, cameras, 
accelerometers, and gyroscopes to track the orientation and rotation of a mobile 
device (Dunleavy et al., 2008). MR is an overarching term to cover the spectrum 
of different technologies, including AR, that to varying degrees, fuse or anchor 
the virtual and the real (Maas & Hughes, 2020). These simulation technologies 
generate an enhanced sense of immediacy, presence (the sense of being there), 
and in some applications, immersion (Bronack, 2011), with emergent potentials 
for literacy and media practices. 

VR also simulates virtual objects, but typically without anchoring the virtual 
to the real world as occurs in augmented and mixed reality environments. VR 
involves immersive, three-dimensional experiences of simulated worlds using an 
HMD and haptic controls with motion sensors to manipulate objects in a com- 
puter-generated, multimodal environment (Velev & Zlateva, 2017). Unlike aug- 
mented and mixed reality platforms, the immediate physical world is completely 
blocked from view to simulate full immersion in the virtual environment (Jensen & 
Konradsen, 2018). This renders the physical or real world barely perceptible in 
the VR experience, only sensed by incidentally touching objects in the play 
space that are concealed from view (Mills et al., 2022). 

VR texts are fully immersive, three-dimensional spaces, enhancing the user’s 
sense of presence, of actually being in the text, with a strengthened degree of 
immediacy. For example, in recent research (Mills & Brown, 2021), researchers 
observed students re-enacting Greek myths using a virtual painting program, 
Google Tilt Brush, mediated by an HTC Vive VR headset. Many of the stu- 
dents were amazed by the sense of immediacy of the historical scenes that they 
imagined and painted. In the students’ words, their ideas literally “came to life” 
and the VR space created “a sense of it’s here” and “it’s real”. As one of the girls 
explained of her Trojan war painting: “You can go towards it”, and “‘it feels like 
it’s an actual world you can be in.” Similarly, one of the boys depicted Icarus’s 
glowing wings projected from the sides of the student’s own body, while the 
form of Icarus was conspicuously absent. This is because the boy imagined him- 
self to be Icarus as he viewed the scene in first person perspective, flying towards 
the radiant sun with the expansive sea below (Mills & Brown, 2021). 
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Virtual and augmented reality textual practices typically involve three- 
dimensional digital imagery, which differs from the plethora of two-dimensional 
textual forms (e.g. web pages, blogs, still images) that circulate on the internet. 
These virtual texts are simulations, and thus objects represented in a virtual 
world have a distinctive kind of virtual materiality that lacks physical tangibil- 
ity and is distinguished from conventional literacy and artistic representational 
media, such as books, pens, or paint and canvas. At the same time, these simu- 
lated, three-dimensional texts and objects typically have multisensory interac- 
tive elements that are responsive to user movement, such as in the case of VR, 
offering different locomotive and haptic engagement and action within virtual 
texts, including locomotion and movement of the whole body throughout the 
virtual space. 


New directions for VR, AR, and MR literacies 


Literacy practices, whether digital or not, whether virtual, augmented, or other, 
can be classified by social context, participation structure, participant roles, pur- 
pose, topic, tone, activity, norms (conventional practices), and language code 
(Herring, 2007; Hymes, 1974). While literacy researchers have referred to waves 
of digital and media literacies practices, sometimes grouped by their affordances 
or enabling features (e.g. Wohlwend, 2017), the distinctive possibilities of virtual, 
augmented, and mixed reality texts in education have not been given sufficient 
attention. Within educational research, these technologies and their applica- 
tion are most frequently researched and theorised in science, technology, engi- 
neering, and mathematics (STEM) education (e.g. Hussein & Natterdal, 2015), 
while their potentials for language and literacy learning are much less frequently 
explored. For example, in a review of original VR studies, only 27 of 167 had 
some relevance to language learning (Reisoğlu et al., 2017). Even when the 
studies address literacy, the educational applications do not typically reflect how 
technologies are used in social sites beyond schooling. For example, early literacy 
skills are often presented through ‘skill and drill’ exercises, such as matching 
games and spelling quizzes, akin to many previous technologies for education 
consumer markets, but with immersive displays and, in the case of MR, a blend 
of virtual and real objects (e.g. Fan et al., 2020). 


Social factors: AR, VR, and MR 


It has long been thought that the medium influences the meaning of commu- 
nication, as reflected in McLuhan’s (1964) phrase “the medium is the message”. 
Yet the significant facets of novel media, immersive VR, and MR have not 
yet been categorised in terms of the technical dimensions that condition these 
emerging practices with social, textual, and communicative potentials. Posited 
here is an original classification system, drawing on a range of empirical studies 
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to explain their inclusion. It is acknowledged that these technologies, while hav- 
ing some common facets of the medium, are also changing, and that other stud- 
ies and future research may contribute additional factors (Mills, 2010). 

In terms of social factors of discourses, these can be readily applied to vir- 
tual, augmented, and mixed reality. For example, any virtual, augmented, or 
mixed reality activity occurs in a particular social context that shapes technology 
use, such as recreational sites, educational contexts, home, or commercial set- 
tings. For instance, medical training programs have experimented with VR to 
simulate surgeries that would be too difficult or dangerous to perform in real 
life (Pottle, 2019); while in museums, young people and families have used 
VR technology to support knowledge of 3D museum objects in relation to 
their historical context (Rae & Edwards, 2016). The social context includes 
networked gaming cultures and communities, commercialisation, and market- 
ing of technologies, ideologies, and worldviews that influence the history and 
design of the technologies. The social factors in any interaction influence the 
literacy practice, conditioning variations in discourses, with many social facets 
operating simultaneously. Social situation facets can be categorised, as indi- 
cated in Table 7.1. 

The notion of participation structures includes the number of active partici- 
pants, turn-taking, setting-shifts, and duration of turns. Currently, VR headsets 
and AR smartglasses — eyeglasses that overlay virtual information over the glass’s 
lens — are mostly designed for individual users in homes or in commercial game 
arcades. Early adopters of technology in higher education are researching large 
group use of these wearables (Birt et al., 2018). Participant characteristics that influ- 
ence the discourse can include factors such as player and observer roles, gender, 
age, dis/ability, culture, social status, experience with the technology, knowledge 
of the content or game, attitudes and motivation, and other attributes, both in 
real life and in terms of virtual personae. 

Other social elements briefly summarised here are purpose (e.g. recreation, 
knowledge, skill development), topic (e.g. narrative, conceptual), tone (formal/ 
informal, serious/playful), activity (e.g. game, navigation, model, simulation, 
book reading), norms (e.g. conventions, game etiquette, online social status), and 
language code. The language code used across a variety of VR, AR, and MR can 
include an array of features, such as the language (e.g. English, Spanish), lan- 
guage variety, dialects, vocabulary, grammatical forms, orthography, fonts, length 
of text, and text presentation (Herring, 2007). Visual grammar incorporates 2D 
and 3D imagery, reading pathways, narrative and conceptual depictions, viewer 
position through representation and interaction, modality and validity, and com- 
positional meanings. Principles of colour from visual design, such as value, sat- 
uration, hue, purity, modulation, transparency, luminosity, and differentiation 
(Kress & van Leeuwen, 2020), are equally applicable to virtual and augmented 
reality environments. 
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TABLE 7.1 Social situation facets of AR, VR, and MR 


1 Participation Group size, e.g., one-on-one, one-on-group 


structure Public or private 

Formal or informal participation structure 
Contribution frequency and quality 

2 Participant Participant age, gender, culture 
characteristics Experience levels 

Roles: in real life/virtual context 


Interests, values, attitudes 


00000000 0 


3 Purpose Social purpose — e.g., professional, recreational, social, 
transactional, communal 

Goal of interaction — e.g., play, learn, create, read 

4 Topic Topic/s of exchanges 
Topic of group interest 
5 Tone Formal/informal 

Attitude/mood conveyed (dis/satisfied) 
Humorous/serious 
Positive/neutral/negative 

6 Activity 


7 Norms 


Play a game, complete a task, visualise, problem-solve 
Social 

Cultural 

Language 

Community 

8 Language code Written language conventions 
Spoken language conventions 
Genres and text types 

Syntax 

Vocabulary and orthography 
Fonts, font sizes, graphics 

2D and 3D imagery 


Visual reading pathways 


9 Visual grammar 


Narrative and conceptual depictions 
Viewer positions 

Representation and interaction 
Modality and validity 


Compositional meanings 


T O pro g GUO nG a T O a S OO a a OO a S OO a aan 


Colour (value, saturation, hue, purity, modulation, 
transparency, luminosity, and differentiation) 


The medium: AR, VR, and MR 


VR, AR, and MR technologies are characterised by a number of technological 
facets that condition the range and nature of social relations (see Table 7.2), and 
these interact with the previously outlined social situation factors. 

The extent to which the medium depends on virtual reality alone, or mixes the 
virtual with the real, influences the social practice. For example, a fully immersive, 
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TABLE 7.2 Medium facets — VR, AR, and MR technologies 


Virtual reality/mixed reality 

Synchronous/asynchronous 

Single user/multiplayer 

Online/offline 

Presence (the extent of presence is influenced by interactivity and realism) 
GPS Location tracking/no tracking (AR only) 

3D/2D 

Moving image/still image 


OMAN D UM FWNEH 


Sensors (e.g. head tracking, eye tracking, haptic, body movement, locomotion) 
Wearable/non-wearable technology (e.g. phone vs. smartglasses, data gloves) 
Channels/modes — visual, audio, haptic, vibration, olfaction, taste, texture 
Ephemerality/permanence 

Pseudonymity 

Voice/chat/messaging 

Distribution/reproducibility 

Materiality/immateriality (e.g. physical book with augmented reality content) 


Anu WONF CO 


virtual experience blocks any view of the real world (Pottle, 2019), rendering 
the virtual content more visually and sensorially focal than the backgrounded 
action of others in the real world, such as teachers or peer observers. In contrast, 
augmented and mixed reality often supports interaction with both virtual and 
real objects or environments, so that participants in the real world are visible. For 
example, a user wearing HoloLens 2 smartglasses (MR) typically sees everything 
in their immediate, real environment, while also seeing a virtual layer of pro- 
jected characters, screens, menus, and objects that are visible only to the wearer 


(see Figure 7.3). 


FIGURE 7.3 Student making 3D Holograms to illustrate an MR story. Ethical con- 
sent was provided for the use of all facial images in the book 
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Whether the communication is synchronous (same time) or asynchronous (dif- 
ferent times) is an important factor in virtual, augmented, and mixed reality 
because it concerns whether or not any interactions will occur simultaneously 
with others who are geographically remote, but virtually connected. For exam- 
ple, used in the context of multiplayer gaming, a VR headset can be used to 
enhance the gaming experience while the user interacts with other players ver- 
bally and socially at the same time (e.g. Vivecraft — multiplayer Minecraft VR). 
This medium creates a very different discourse than a virtual experience that 
is recorded and shared to be played back or recorded with other synchronously 
engaged participants to be replayed by others at a later time (asynchronous). 

Virtual, augmented, and mixed reality media offer single user or multiplayer 
modes (two or more). The affordances of each mode influence the range and 
nature of interaction, the facility for collaboration, or pursuit of mutual goals, 
all of which will influence the social practice. For example, students can play 
Minecraft in VR with other students who are wearing a VR headset in another 
location, collaborating in real time to build, explore, or conduct virtual transac- 
tions in the game (https://www.minecraft.net/en-us/vr). 

Synchronous voice communication can be used to play together if connected 
using VOIP (voice over internet protocol — calls over the internet), such as with 
Discord, a free voice, video, and text application (https://discord.com/). Often, 
VR games are conducted in single user mode, without the ability to collaborate 
with another user. Of course, users who are playing a virtual game in the same 
room can take turns using the headset, while participating and observing the 
action on the computer screen. 

Online and offline applications of VR, AR, and MR differ in terms of how 
interactions occur with the technology, most critically because the functional- 
ity of various social applications may be dependent on internet connectivity. 
For example, an offline VR game, like Vive Spray 2, will involve interaction 
with downloaded computer-generated content, such as spray painting a virtual 
subway station with graffiti art — a solitary artistic and creative game — rather 
than involving interactions with other players (https://store.steampowered.com/ 
app/614830/ViveSpray_2/). Contrastingly, Google Earth VR involves online 
connectivity to the internet to view geographically locations around the globe 
in a more immersive way than on a screen. Wooorld now allows users to explore 
the world in VR with friends located elsewhere (https://www.wooorld.io/). 

The extent to which virtual, mixed, and augmented reality technologies sup- 
port the users’ psychological sense of presence, of actually being there (Heeter, 1992), 
is an important facet of the medium that conditions the social practice. A sense 
of presence is the general feeling and mental orientation that one is having a real 
experience, which may or may not be experienced as real sensorially. Interactivity 
and realism are two key factors that contribute to this sense of presence (Von Der 
Piitten et al., 2012). Interactivity, the responsiveness of digital technology to the 
user’s input, can influence the sense of presence. For example, playing a VR game 
creates a stronger sense of presence than simply observing a game (Vorderer et al., 
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2009). Realism can include factors such as the resolution of the graphics, whether 
the events are in first-person or third-person view (Hoffmann et al., 2010), the 
realism of the player’s virtual form, and the degree to which virtual objects behave 
as they would in the real world (Blascovich et al., 2002). 

Location-based or geo-based AR uses a GPS, digital compass, and accelerometer to 
detect or tag precise locations, including the user’s position. For example, the enter- 
tainment industry has utilised location-based AR gaming in popular applications, 
Minecraft Earth and Pokémon Go, which involve placing virtual objects in precise real- 
world locations, with potentials for viewing and interaction by others who traverse 
the same public spaces (and within the particular privacy and authorisation settings 
of the game). While the social purpose for location-based AR applications varies 
significantly (e.g. educational, commercial, productivity, entertainment), this facet of 
the medium conditions the types of interactions possible among users, with virtual 
content anchored to particular geolocations that are accessible by physical proximity 
in the real world. For example, Wonderscope uses AR to enhance reading experiences 
for kids (https://wonderscope.com/), where stories like Clio’ Cosmic Quest allow 
children to move around the physical world while interacting with Clio who talks 
audibly and directly to the viewer (see Figure 7.4). The child reads and interacts with 
words that appear on the screen to make choices in the narrative. 


FIGURE 7.4 Augmented reality story, Clio’s Cosmic Quest in the Wonderscope app 
produced by Within Unlimited Inc., overlaying virtual objects on the real world 
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Whether the virtual or mixed reality environment is presented using 2D or 3D 
visual representations or a combination of both, will influence the social prac- 
tice because three-dimensionality often invites different reading paths to two- 
dimensional displays. For example, three-dimensional scenes in immersive VR 
can involve the user walking to different areas within a play space. Viewing vir- 
tual objects in a 360-degree space involves moving the body and head to explore 
and shift one’s gaze around objects, to see the front, sides, and back. In contrast, 
viewing a two-dimensional image can occur from a single vantage point from the 
front (Kress & van Leeuwen, 2020). While most virtual and mixed reality envi- 
ronments involve some three-dimensional elements, the quality of the graphic 
displays vary, and two-dimensional representation is not uncommon, such as 3D 
virtual scenes or objects that can be translated and shared online in a 2D format. 
There are very different meanings for social interactions that are conditioned by a 
sense of presence (Mills & Brown, 2021), different forms of sensory engagement 
and physical movement within the text (Mills et al., 2022), and various spatial 
orientations within a three-dimensional virtual or mixed reality environment. 

Social practices using virtual, augmented, and mixed reality technologies may 
involve still or moving imagery, or combinations of these, which conditions the 
potential meanings. Moving texts, such as those that appear in filmic media, 
video, television, and many contemporary video games, are sometimes referred 
to as kineikonic texts. The systems of signification in kineikonic texts, such as 
image (including dramatic action, gestures, facial expression, etc.), narration, 
music, or even silence, are combined in multimodal ways to create meaning 
within the spatial and temporal frames of filming and editing (Burn, 2013; Mills, 
2011). The representational, interactive, and compositional meanings of kinei- 
konic or moving image texts have been analysed elsewhere by social semioticians 
who note that action and transactions are realised by movement, rather than vec- 
tors, as occurs in still images (Kress & van Leeuwen, 2020). 

The use of sensors is an important facet of virtual and mixed reality technolo- 
gies which shapes the social practice. In the case of immersive VR technology 
with HMD, a positional tracking system uses infrared light-emitting base stations 
to communicate with sensors worn by the user to create 360-degree, room-scale 
virtual immersion. The sensor-based tracking system enables a sensorial form of 
computer-human interaction that simulates embodied interactions in the real 
world (see Figure 7.5). 

When used to explore historical and literary content, for instance, such embod- 
ied interaction with historical times and places has been observed to build the user’s 
empathy with literary characters (Moran & Woodall, 2019). Mixed and augmented 
reality applications may use one or more motion-tracking systems, such as digital 
cameras or other optical sensor, GPS, radio-frequency identification (RFID), gyro- 
scopes, solid-state compasses, or accelerometers. For example, in research of AR 
technology, users’ social connectedness and non-verbal behaviour were influenced 
by the presence or absence of a headset in a group setting (Miller et al., 2019). 
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FIGURE 7.5 Using HTC Vive VR sensors 


The extent to which VR, AR, and MR activities involve the use of wear- 
able technology influences the social meaning, and there is a growing variety of 
wearable devices, from AR smartglasses to VR HMD that block out views of 
the real world (Jensen & Konradsen, 2018). Other VR technologies are not 
wearable, such as immersive rooms, or smartphones that support AR. The social 
conditions of each vary in particular ways. For example, when a user wearing 
smartglasses concentrates their gaze on virtual information, people nearby are 
effectively ‘looked past’ and ignored. 

Virtual, augmented, and mixed reality platforms combine multiple channels of 
communication, often referred to as modes (Kress, 2000). These media typically 
give priority to vision, which may include the visualisation of written words, 
images, spatial layouts or graphic formats, and gestural meanings. The visual 
channel may or may not be supported by eye-tracking technology and sensors 
to guide vision through head movement. Audio, such as background music, is 
typically supported by VR platforms, with headphones and microphones in the 
HMD, while AR apps for education, commercial use, or productivity, such as 
those that overlay 3D images through one’s smartphone camera, may background 
(e.g. music, sound effects), foreground (e.g. virtual character speaking directly to 
the user), or exclude sound. Each offers different semiotic resources for com- 
munication, with each mode carrying a different functional load of the meaning. 

VR technologies often track the user’s body movements via a range of 
motion-sensing technologies, with movement being an important channel for 
making meaning. Other channels for simulated communication include tactility 
(user action), vibration (user receives), or airflow (user produces into a micro- 
phone) through various game controllers. Channels to simulate olfaction, taste, 
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and texture have been developed to enhance narrative meanings in game play. 
For example, the taste of food has been simulated through electrodes on the 
tip of the tongue (Ranasinghe & Do, 2016), while the texture of eating food 
has been simulated through electrodes that stimulate the masseter muscle in the 
jaw (Niyjima & Ogawa, 2016). Examples of how taste has been used to support 
meaning making across a range of media were discussed in Chapter 5. The tra- 
jectory of development in human—computer interaction is rapidly opening up a 
greater number of sensorial channels for simulated communication, experience, 
and interaction. 

The degree to which virtual, augmented, or mixed reality practices are char- 
acterised by temporality or permanence may influence the social dimensions of the 
practice (Herring, 2007). For example, Snapchat, which overlays AR filters and 
art over selfies, automatically deletes digital content sent by someone after it is 
viewed rendering it temporal. Texts that are shared and permanently viewed by 
others on social media (e.g. Instagram) have a different social function than a 
transient or ephemeral text, such as taking an AR image to see how a potential 
purchase might look in a home. 

In addition, the extent of user anonymity or pseudonymity is relevant to virtual 
and mixed reality technologies with direct implications for how one interacts, 
pretends, or edits one’s digital persona (Herring, 2007). Immersive VR game 
play often involves the user selecting a player name that is different to one’s real 
name or taking on a game persona using a pseudonym. Anonymity involves 
the absence of any revelation of personal information or identifiers. Research 
of other online interactions has shown that anonymity increases the frequency 
of self-disclosure (Kiesler et al., 1984), role play (Danet, 1998), and antisocial 
behaviour (Donath, 1999; Herring, 2007). This does not necessarily mean that 
the user cannot be identified to technology companies. 

Virtual, augmented, and mixed reality technologies may include integrated 
facility for messaging, chat, or voice call, or require a separate program. This might 
occur in the context of a multiplayer game while using HMD with built-in 
microphone to talk to another player. VRChat is an example of a massive mul- 
tiplayer online VR game where users appear as 3D models, and speak to one 
another socially in groups, or in separate two-person conversations, configuring 
the discourse in particular ways (VRChat, 2020). 

Technology support for the reproducibility and distribution of digital content 
varies across VR, AR, and MR, which often enables users to save and share ele- 
ments of their virtual experience. For instance, users may save images produced 
in an AR app to a smartphone’s image library for easy distribution on social 
media. In contrast, VR video recordings, such as Google Tilt brush’s Tilt files, can 
be converted into an accessible format to be viewed by those without Tilt brush 
software (see Figure 7.6). Thus, the ease of distribution of augmented and virtual 
reality texts varies across platforms and applications, with differing emphasis on 
enjoying the social experience itself or sharing about the experience with others. 
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FIGURE 7.6 Boats in Trojan War painted by a student using Google Tilt brush 


The materiality or immateriality of virtual, augmented, and mixed reality texts, 
hardware, and software varies substantially, influencing the social conditions of 
their use. Immersive, VR technologies project a virtual environment or ‘text’ that 
is visible, but which is essentially immaterial, characterised by a lack of corporeal- 
ity or tangibility. For example, Smartglass technologies have built-in processors, so 
that the relatively lightweight glasses can be worn in various social settings while 
on the move or around the classroom (Ibrahim & Ali, 2018). Many mixed and 
augmented reality technologies support hands-on interactions with both physical 
learning materials, such as tangible books with links to AR content, flashcards, and 
pencils, with virtual content displayed on a tablet or smartphone, which orientates 
the social interaction in particular ways (Fan et al., 2020). 

The purpose of the current classification was to identify and systematise the 
key dimensions for VR, AR, and MR literacies that influence interactions with 
others and virtual texts. It is formulated on the understanding that digitally 
mediated literacies are shaped at the most elementary level by the social situation 
of use and the medium (Herring, 2007). Discourse norms will not be the same 
for different social purposes and virtual environments. 


Tensions for VR, AR, and MR literacies technologies 


While VR, AR, and MR platforms are changing the nature of literacy and com- 
munication practices in definitive ways, particularly in commercial and recre- 
ational spaces, the full extent of these transformations is yet unseen. In 2018, the 
number of people who had used an AR application at least once a month in the 
USA was 59.5 million, forecast to exceed 95 million by 2022 (Vailsherry, 2021). 
AR applications have been associated with improved kinaesthetic, spatial and 
visualisation skills, critical thinking, and active involvement in learning — skills 
that are supportive of language and literacy practices (Altinpulluk, 2019). 
Likewise, VR technologies have become more common in education, with fully 
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immersive meaning-making potentials that include virtual painting, manipulat- 
ing three-dimensional models, reading interactively, and many other ways of 
using information and sign-making (Dooley et al., 2020; Mikelli & Dawkins, 
2020; Rose, 2018). 

The commonly cited constraints for language learning have included the ten- 
dency of these platforms to prioritise visual information (Mills, 2016), the limited 
and inflexible range of education content developed to date, and the difficulty 
for teachers to develop and adapt applications for their own purposes. Others 
have noted technical difficulties in classroom settings, such as unstable or imperfect 
tracking (Fan et al., 2020). Some AR and VR literacy applications are based on 
a narrow definition of literacy as phonological and word recognition skills, with 
many literacy games based on behaviouristic, stimulus-response quizzes and factual 
recall (Fan et al., 2020). Likewise, virtual and mixed reality technologies are subject 
to some of the common barriers to teacher take-up of any new technology in the 
curriculum, including insufficient professional development for teachers, teach- 
ers’ lack of time, an overcrowded, and in the case of some VR and MR systems 
(e.g. HMD, smart glasses), expensive computer upgrades, maintenance, and space 
requirements (Chandra & Mills, 2014; Sirakaya & Sirakaya, 2020). 

One of the foreseeable tensions for the use of extended reality technologies 
for language and literacy learning will be closer work between educators and 
technology developers to ensure that these platforms are suitable for curriculum 
use in large and small group learning contexts. This requires technology devel- 
opment research in which these new media are researched and adapted in situ for 
classroom use, with input from literacy educators and curriculum developers to 
guide the future of virtual, augmented, and mixed reality technology for positive 
educational change. 

The necessary development of assessment tasks and rubrics to ensure that these 
technologies are not marginalised in the curriculum is anticipated to be an addi- 
tional challenge, given that print-based literacy practices dominate standardised 
testing programs on a global scale. In addition, given the digital divide in-home 
computer and internet access (see Warschauer & Tate, 2018) — technologies that 
are needed to support many AR, VR, and MR applications — it is important for 
schools to provide access to these newer learning technologies for purposeful 
learning gains that will otherwise become the exclusive domain of the middle 
class. Within the context of current growth of extended reality technologies, AR 
applications that are supported by mobile phones have become one of the most 
widely accessible technologies to the masses (Sommerauer & Miiller, 2014). 


Implications of VR, AR, and MR for literacy curriculum and 
pedagogy 

While research of VR and MR applications for language in education currently 
only constitutes a small proportion of published work (Reisoglu et al., 2017), 
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there are clear potentials for using VR, AR, and MR for the literacy curriculum. 
Emerging potentials of these hybrid technologies have been identified for speak- 
ing and social learning, language code breaking, reading and comprehending 
texts, and creative digital text production. Spoken language learning is a vital 
foundation for literacy practice, so it is interesting that virtual and mixed reality 
technologies have been found to provide a supportive environment for speak- 
ing and social learning. For instance, some people with introverted personali- 
ties report feeling more comfortable in virtual social interaction compared to 
face-to-face social setting (Brennan, 2017). Additionally, researchers have found 
learning gains for children’s AR-supported dramatic play (Han et al., 2015). 

In relation to VR, AR, and MR technologies for literacy, there are numerous 
studies that demonstrate the potential for teaching language code breaking using 
games-based pedagogies (Fan et al., 2020). Most notably, a number of studies 
have explored AR technology games for teaching spelling (Pu & Zhong, 2018), 
word recognition, word and object matching (Barreira et al., 2012), sound and 
object matching (Fan et al., 2020), and other collecting games, usually in location- 
based educational AR applications (Hsu, 2017). 

Extending beyond phonics and other early literacy skill development, 
emerging VR research has considered the potentials for creative digital text 
production. For example, Mills and Brown’s (2021) research with open-ended 
design software, Google Tilt brush, was used with upper elementary students 
in generative ways to transmediate or shift meanings from conventional writ- 
ing and drawing in two dimensions, to three-dimensional immersive textual 
creation across multiple modes. The lack of equivalence between sign sys- 
tems (2D drawing to 3D VR painting) established an anomaly for the learners 
that led them to invent new ways to represent concepts three-dimensionally. 
Students used amplified haptics and locomotion to create ‘surround images’, 
with immersive effects, conjuring a sense of presence in their immersive digi- 
tal designs that was distinct from conventional drawings and writing on paper 
(Mills & Brown, 2021). 

Recent research has pointed to the potentials of VR, AR, and MR technolo- 
gies for reading, comprehending, and making meaning from texts. For example, 
in classroom research by Moran and Woodall (2019), teachers used VR headsets 
to support the students’ interpretation of the novel, Tò Kill a Mockingbird. The 
students were able to walk around a 360-degree virtual space to explore the 
historical context first-hand, which enabled the students to empathise with the 
experience of the characters. Further, students were able to think critically about 
narrative settings, time, and place in literature more broadly. Another example 
is AR interactive books, which contain overlaid virtual content, such as three- 
dimensional models, viewed through a mobile device (see Chapter 10). Interactive 
meaning-making activities based on the reading of an AR book can involve tasks 
such as organising events from the narrative in chronological sequence, support- 
ing meaning making with books in interactive ways (Vate-U-Lan, 2011). 
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Recommendations for research of VR, AR, and MR literacy 
practices 


While virtual, augmented, and mixed reality technologies have generated new 
textual formats that combine visual, tactile, audio, and other channels that are 
only emergent in media and literacy research, technology developers are still 
exploring the simulation of touch, taste, and smell as human—computer interac- 
tion modalities, which are currently missing in most VR and AR technology 
platforms marketed on a mass scale. Technology researchers envisage a future 
in which the expansion of virtual, augmented, and mixed reality platforms for 
everyday users is more fully multisensory and immersive, stimulating more than 
vision and hearing in immersive virtual environments (Mills & Friend, 2021). 
Likewise, the expansion of communication channels in virtual and augmented 
simulations will contribute in some ways to the development of more inclusive 
digital environments for the deaf or blind, working against ableist assumptions 
that are complicit in technology development, and the Western, empiricist pri- 
oritising of the visual mode or observation as the main source of truth, and as a 
higher form of knowing (Mills et al., 2018). 

Researchers who apply a human-centred approach to guide these technology 
developments similarly point to a lack of vocabulary or nomenclature to describe 
olfactory, tactile, and gustatory experiences in the same granularity as visual and 
audio elements to guide human-computer interaction development (Anjum, 
2019). For example, the language of taste focuses on broad descriptors such as 
sweet, sour, and salty, yet if electrodes on the tongue are to simulate the varieties 
of human taste sensations, both the nomenclature and simulation technology will 
need to advance beyond three main experiences. In the future, it is anticipated 
that virtual, augmented, and mixed reality technologies will have higher simula- 
tion fidelity, that is, the extent to which the simulation mirrors interaction in the 
real world (Hamstra et al., 2014), with texts that require the use of an expanded 
sensorium that includes visual, auditory, tactile, gustatory, and olfactory features 
(see Chapter 5). Narrow, text-image-based conceptions of sign making will need 
to be expanded to account for odours and flavours that have been invested with 
meaning, such as the use of the aromas of incense in many religious traditions to 
denote the prayers of the faithful, or the flavour of cinnamon as a status symbol 
in the European Middle Ages. 

Research on embodiment and learning demonstrates that the precise ways in 
which the body is mediated by digital tools actually matter to the mind, to mem- 
ory, and to understanding, whether it be the way visual attention is more focused 
when writing by hand compared to keyboarding or how handwriting speed, an 
often taken-for-granted fine motor skill, is actually a strong predictor of students’ 
ability to compose quality, lengthy texts (Haas & McGrath, 2018). It comes 
as no surprise then, that virtual, augmented, and mixed reality technologies, 
with their varied material and immaterial forms, hybrid sensorial interaction, 
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motion-tracking, and location-based features, position the body in different ways 
that matter for the mind, for literacy, and for the representation of knowledge. 
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PART III 
Texts and digital semiotics 


Technology is fundamental to the semiotic, or meaning-making, processes in 
which we engage through our creation and interpretation of texts expressed in 
various forms of media. The materiality of texts and the nature and extent of our 
bodily engagement in interpreting and producing them have always been related 
to the technologies for communicating meaning that are available in society. The 
semiotic affordances of different types of communication technologies favour or 
facilitate different kinds of meaning. Kress and van Leeuwen (2020, pp. 227—228) 
distinguish three classes of such technologies according to the ways in which 
they produce representations of meanings: (i) those articulated by the human 
hand, aided by handheld tools such as brushes, pencils, keyboard, mouse, etc; 
(ii) audio and visual recording such as digital photography and movie-making, 
“which allow more or less automated analogical representation”; and (iii) tech- 
nologies which allow digitally synthesised models of the phenomena to be rep- 
resented. As Kress and van Leeuwen (2020) point out, the boundaries between 
these categories are not clear cut. The chapters in this section evidence a range 
of combinatorial applications of these categories of technologies reflecting the 
changing materiality of texts, differing bodily engagement in their creation and 
interpretation, and the resultant expansion of semiotic choices in multimodal 
digital age texts. 

In Chapter 8, we discuss the increasing prominence of infographics, partic- 
ularly in public communication of health and environmental information but 
also in science and technology curricula in schools (Unsworth, 2020, 2021). 
Today, we readily recognise infographics as page or screen-sized image—language 
ensembles that usually include at least one image of various kinds such as photo- 
graphs, diagrams, graphs, maps, cartoons, etc. separately or in combination, along 
with text in the form of image captions, ‘call-outs’, annotations within images, 
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and interpolated text blocks. Such contemporary infographics combine hand 
articulated and recording communication technologies. Their communicative 
role entails the condensation of complex, detailed information as visual-verbal 
‘synoptic eyefuls’ within the spatial confines of one page or screen. The effec- 
tive bi-modal portrayal of meanings in this format requires not only astute and 
appealing design but also dextrous creation and manoeuvring of a variety of 
representational forms. Mapping the available designs for infographics provides 
a metalanguage describing the meaning-making options they afford, which can 
inform the development of critical infographic literacy. Chapter 8 outlines cur- 
rent approaches and future possibilities along this trajectory. 

The earliest forms of animation involved devices such as a spinning card- 
board disk of slightly different images that created the illusion of movement 
when viewed in a mirror. These animations relied entirely on hand-articulated 
technologies, but these were augmented in later forms of animation with record- 
ing technologies that enabled filmic animations. This sequential augmentation 
of different production technologies persists as stop-motion animation but has 
become increasingly inter-related in digital animation creation that integrates 
computer programming with multimodal animation authoring. In Chapter 9, 
we discuss primary and secondary school students’ involvement in this form of 
animation creation through coding animated narratives. The physicality of this 
kind of multimodal digital text creation cannot be underestimated. The kinds of 
meanings that can be conveyed rely on students’ dexterity in modifying digitally 
drawn avatars to represent different feelings or actions — or drawing their original 
avatars — as well as their physical construction of the computer program (in our 
examples by assembling on-screen virtual blocks representing computer codes). 
This coding results in the dynamic portrayal of the story. Recording technolo- 
gies enable the student authors to incorporate their own or others’ voices for the 
characters and to include original or selected sounds and music as part of the 
animation. In Chapter 9, we discuss existing pedagogic approaches to coding 
animated narratives and suggest a re-configuring of research and pedagogy at the 
intersection of coding and multimodal authoring as a basis for further advancing 
this aspect of literacy for digital futures. 

Kress and van Leeuwen (2020 p. 228) indicated that synthesising communi- 
cation technologies are not only tied to recording technologies oriented to the 
eye and ear, but in addition reintroduce hand articulation technologies through 
interfaces such as the keyboard and mouse, and increasingly through direct artic- 
ulation of the body via spoken commands or responses to the computer, and 
through touchscreens and the manipulation of digital devices such as tablets. The 
materiality of contemporary and emerging digital interactive literature exempli- 
fies these synthesising technologies, which are creating new dimensions of liter- 
ary experience through new forms of literate practice and expanding the nature 
of bodily engagement in literary story worlds. Chapter 10 examines the nature 
of user bodily interactivity in works of digital interactive literature for children 
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and adults that involves interacting with fictional entities through activity such as 
screen swiping or device shaking in animated picture books, and confronting the 
seeming corporeality of characters and materiality of settings in picture books, 
short stories, and novellas through various forms of virtual reality apps. A frame- 
work is developed that maps complex networks of options for interactivity to the 
multiple meaning-making dimensions of language and image to inform ongoing 
research and to derive future-oriented pedagogies that will address new ways of 
experiencing and interpreting digital multimodal literary narratives. 

Core to developing literate competence, whatever communication technolo- 
gies and representational modes are involved, is the negotiation of meaning. This 
negotiation entails an appreciation of the three simultaneously occurring, distinct 
and inter-related dimensions of meaning: ideational meaning referring to the rep- 
resentation of events, participants, and circumstances in experience; interpersonal 
meaning referring to the nature of relationships among participants; and textual 
meaning, pertaining to the relative information value and cohesion among the 
represented elements (Halliday, 1978). But negotiating meaning also entails under- 
standing how these dimensions are communicated through various modes. This 
requires a metalanguage describing the options for the expression of meaning in 
the different modes and their multimodal combinations (Rose, 2020a, 2020b). 
Pervading this section is our contribution to advancing this evolving metalan- 
guage as a resource for pedagogies seeking to develop literacy for digital futures. 
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INFOGRAPHICS AND SCIENTIFIC 
LITERACY 


Introduction — infographics in 21st-century literacies 


The digital multimedia communication world of the 21st century, with its pre- 
dominant information-based economy, is characterised by cultural and societal 
expectations of increased engagement with the readily accessible and ongoing 
proliferation of information in all walks of life. The promotion of ubiquitous 
popular consumption of information in these varied realms has led to an ever- 
increasing emphasis on visualisation, and the notion of communicating large 
amounts of often complex information as an ‘eyeful’ within single page or screen 
size information graphics. These infographics are image—language ensembles 
that usually include at least one image of various kinds such as photographs, 
diagrams, graphs, maps, cartoons, etc. separately or in combination, along with 
text in the form of image captions, ‘call-outs’, annotations within images, and 
interpolated text blocks (Unsworth, 2021). They have been characterised as hav- 
ing the capacity to grab readers’ attention, and to condense, connect, and make 
accessible components of high volume, complex issues (Conner, 2017; Damman 
et al., 2018; Dunlap & Lowenthal, 2016). 

Infographics have been shown to have aesthetic appeal to a broad general 
readership (Lima, 2011, 2014, 2017). They are also increasingly being used in 
professional journals such as the British Medical Journal (BMJ, 2021). The BMJ 
website states: 


Our infographics are a new initiative at the BMJ. We know that you, our 
readers, are increasingly time-pressured, so we aim for them to include 
some carefully selected information from an article, highlighting the key 
messages.... BMJ infographics are reviewed by the authors, our technical 
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and section editors and with some the peer reviewers before publication, 
and we’re confident that they are accurate representations of the article. 
(BMJ, 2021) 


But infographics are used extensively and are seen as increasingly important in 
communicating to the general public on websites and in the popular press, as 
well as in brochures and other publications by government and semi-government 
authorities and industry (Gebre, 2018; Lee & Kim, 2017; Naparin & Saad, 2017; 
Yarbrough, 2019). 

While the occurrence of various forms of infographics can be traced back for 
centuries, there has been an upsurge and widespread proliferation of their use in 
the 21st century (Gebre, 2018; Yarbrough, 2019). Infographics are ubiquitous 
in communicating about a wide range of issues of personal, social, and global 
significance that warrant urgent attention. The importance of infographics in 
educating the general public has been evident in their widespread deployment 
by the World Health Organisation, and by national and state governments and 
health authorities during the COVID-19 pandemic and emphasised by intense 
research on their use (Cavazos et al., 2021; Domgaard & Park, 2021; Jacob, 2020; 
Li & Molder, 2021). Many infographics concerning health and environmental 
issues are focussed on developing a scientific understanding of complex concepts 
in highly condensed, single page formats that provide accurate information, and 
at the same time are visually appealing and understandable by a lay audience. The 
ubiquity of this compact, multimodal form of public education and the signifi- 
cance and urgency of the issues being addressed make infographic representation 
in scientific literacy pedagogy crucial on the agenda for educational research and 
classroom practice. 

In this chapter, we firstly show examples of infographics in science learning and 
assessment that students need to negotiate in schools. We outline the distinctive 
inter-relationships of image and language in these highly condensed but trac- 
table portrayals of complex ideas and indicate the requirements for students to 
create approximations to these kinds of infographic representations in assessment 
tasks. In the next section, we examine research into the co-articulation of language 
and image in student-created infographic representations. Here, we identify the need 
for this research to draw on a more comprehensive and consistent account of 
mechanisms for linking image and language, the variety of text formats used 
in conjunction with images, and the semiotic means by which meanings are 
condensed in the disciplinary discourse of science. In the following section, 
we provide a set of network-mapping options for image—language integration and 
meaning aggregation in infographics. From this mapping, we make some sugges- 
tions for feaching—learning experiences to enhance infographic literacy development, and 
we conclude the chapter with recommendations for researching infographic 
literacy for digital futures. 
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Infographics in science learning and assessment 


In contexts of schooling, the trend toward infographic style representation has 
been documented in science education with the increasing dominance of images 
as the rhetorical locus in ‘trade’ and textbooks (Bateman, 2008; Bezemer & 
Kress, 2010; Kress, 2005; Peterson, 2016). In some cases, this trend has pro- 
gressed to the extent that traditional ‘running text’ has been elided in favour 
of image-based portrayals that include annotations, interpolated text blocks, 
and a caption (Danielsson & Selander, 2016; Martin & Rose, 2012; Unsworth, 
2020a). Figure 8.1 shows a page from a Year 10 science textbook widely used in 
Australian Schools (Chidrawi et al., 2013, p. 19). Under the heading of “Mitosis” in 
Figure 8.1, there are two paragraphs of ‘running text’ and then the infographic is 
designated as “Figure 1.13”. This consists of the caption, the annotated diagram 
of the cell at different stages of mitosis, and the table of dot points whose head- 
ings correspond to the stages of mitosis depicted and named in the diagram with 
each column of the table directly below the corresponding stage in the diagram. 

The running text firstly defines mitosis as “the process in which somatic (body) 
cells undergo a single nuclear division, giving rise to two genetically identical 
daughter cells”. The remainder of the running text indicates the importance of 
mitosis for growth, replacement of damaged cells, and maintenance of an organ- 
ism, as well as historical information about the biologist Walther Flemming’s 
discovery of the processes of mitosis. It is important to note that the brief tra- 
ditional running text does not explain the process of mitosis, and the textbook 
relies entirely on the infographic for this explanation. This is also the case with 
other similar textbooks (Lofts, 2015, p. 28; Silvester, 2016, p. 10). Students need 
to read and interpret each of the diagrammatic representations and the changes 
from one representation to the next, coordinating this visual interpretation with 
the annotations in each case, and then they need to relate this to the table of 
dot points below, which provide additional information to that included in the 
diagram and annotations. This kind of complex multimodal interpretive reading 
practice which involves continuous inter-relating of multiple visual representa- 
tions and interpolated annotation in the absence of traditional main running 
text, is also what is required in reading the kinds of infographics that are used in 
public education campaigns, such as that shown in Figure 8.2 disseminated by 
the National Oceanic and Atmospheric Administration of the US Department of 
Commerce (2021). Hence, developing this form of scientific literacy is important 
learning in school and beyond. 

The importance of infographics in school education is also evident in their 
increasing inclusion in high stakes student assessment programs. Figure 8.3 
is an example of an infographic from the Program for International Student 
Assessment (PISA) widely administered in over 80 countries every three years 
to 15-year-old students through the Organisation for Economic Co-operation 


154 Texts and digital semiotics 


A different take on hotting things up? 


f 


result in all female offspring. 


[M 


In some reptilian species, sex is determined by environmental 
temperature. Cooler temperatures during a critical stage of some turtle egg 
development will result in all male offspring. Warmer nest conditions will 


Mitosis is the process in which somatic (body) cells undergo a single nuclear division, giving 
rise to two genetically identical daughter cells. Mitosis is important for growth, for replacing 


damaged cells and for the maintenance of an organism. 

German biologist Walther Flemming (1843-1905) stained cells taken from an amphibian. In 
the nuclei, he saw thread-like structures he called chromosomes (from the Greek word chroma, 
meaning ‘colour’). The chromosomes divided lengthwise and separated equally between the two 
daughter cells. Flemming named this process mitosis (from the Greek mitos meaning ‘thread’), 
and showed how the cell nuclei of the two daughter cells would be exact copies of the original 


nucleus (Figure 1.13). 


Pairs of centrioles* 
separate and give rise 
to the spindle fibres. 


Cell membrane 
Cytoplasm 

Nuclear membrane 

disappearing 


Prophase Metaphase 


*Centrioles are minute rod-ike structures 
that form the poles of the spindle. 


Prophase (Greek pro 
meaning ‘beginning’) 


Chromosomes coil and © Chromosomes are 
become shorter and visible. positioned around the 
The nuclear membrane 
disappears. 

A network of spindle fibres 
forms. 

Spindle fibres attach to 
the centromere of each 
chromosome. 


ISBN 9780170231510 


Spindle fibres 


Chromosomes 
line up along 
midline of cells. 


Metaphase (Greek meta 
meaning ‘after’) 


equator, forming a circle. 


r 
Aa 


Sex as a game of ‘chance’ | 


Values and issues 


E 


Modelling mitosis 


Y Figure 1.13 
The stages of mitosis were 
first described in detail 
by Walther Flemming in 
1882. By the end of this cell 
division, the two daughter 
cells have the same number 
of chromosomes as the 
original parent cell. 


Chromosomes 
uncoil and become 
less visible. 


Cell eventually 
divides into two 
after cytokinesis 


Chromatids OPEB 
separate and move Nuclear 
to opposite ends of the cell. membrane 
re-forms. 
Anaphase Telophase 
Telophase (Greek telos 
meaning ‘end’) 


Anaphase 


The spindle fibres contract 
towards the poles of the cell. 
The centromeres divide — the 
chromosomes are now two 
sister chromatids. 

Sister chromatids move 
apart. 


FIGURE 8.1 Infographic in a Year 10 science textbook 


Chidrawi et al., 2013, p. 19 


The chromosomes reach the 
opposite poles of the cell. 

The nuclear membrane re-forms, 
and the chromosomes become 
less visible. 

Cytokinesis begins — animal cells 
‘pinch in’ and plant cells grow new 
cell membranes and cell walls 
between the two nuclei. 


19 


and Development (OECD, 2010, p. 100). Here, there are only two lines of tra- 
ditional running text, but students need to coordinate their interpretation of the 
inter-relationships among four different kinds of visual representations, along 
with their associated annotations and interpolated text blocks. 

Similar infographic formats are included in national high-stakes student 
assessments (see, for example, the final year high school public examinations 
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in biology under the New Zealand Qualifications Authority, 2021). But stu- 
dents are also required to create infographic representations in school assessment 
regimes. In senior high school science in Australian schools, students routinely 
complete assessment items based on those that appear in the public matricula- 
tion level examinations. As part of a study of multimodal literacy in senior high 
school, science infographic responses to such assessment tasks were examined 
(Unsworth, 2021). The following is an example of an assessment item from 
chemistry: 


When an experimenter adds 50g of solid sodium nitrate (NaNO3, an 
ionic salt) to 100ml room temperature water it will all readily dissolve. 
Explain how this occurs with reference to the bonding between particles. 
Use a diagram to aid your explanation. 

(Unsworth, 2021, p. 10) 


The student response in Figure 8.4, created under examination conditions, 
includes similar multimodal meaning-making resources to those in the profes- 
sionally published infographics. It shows the effective inter-relationship of two 
diagrammatic representations, along with articulated annotations of different 
kinds and two strategically positioned text blocks. 

However, the response in Figure 8.4 was created by a higher achieving stu- 
dent, while many students in this and other studies in biology and chemistry 
were not able to optimally articulate images and language in multimodal repre- 
sentations (McDermott & Hand, 2013, 2016; Unsworth, 2021; Unsworth 
et al., 2022). 
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FIGURE 8.4 Year 11 student-created infographic response to an examination question 
Unsworth, 2021, p. 13 
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Co-articulation of language and image in student-created 
infographic representations 


In recent years there has been substantial research interest in enhancing science 
learning through student-created infographic representations (Davidson, 2014; 
Fowler, 2015; Fuhrmann et al., 2018; Gebre, 2018; Gebre & Polman, 2016; 
Hand et al., 2016; Polman & Gebre, 2015; Tytler et al., 2018; Walsh & McGowan, 
2017). Some studies emphasised the heuristic value of infographics reporting 
high student motivation and positive impacts on assessment results. However, 
the few studies that included attention to students’ co-articulation of language 
and image did not seem to be informed by any systematic and comprehensive 
account of approaches to establishing cohesive relations between language and 
image (Bateman, 2014; Unsworth & Cleirigh, 2009), or of the strategic use of 
converging or complementary meaning making through language and image. 

The integral role of visualisations in science discourse and education has been 
long acknowledged (Gilbert & Treagust, 2009; Treagust et al., 2017; Treagust & 
Tsui, 2013), but infographics are a nascent genre in research on learning from 
and with visualisations. There has been limited attention to the multimodal 
rhetoric of infographics in research or classroom implementation. In view of the 
paucity of research literature to inform the design process about ways and means 
of reading and creating ‘quality’ infographics, research by Gebre and Polman 
(Gebre, 2018; Gebre & Polman, 2016; Polman & Gebre, 2015) engaged high 
school students in examining and critiquing published infographics. This was 
a precursor to the students generating their own infographics to communicate 
their investigations of self-selected topics, such as cloning, shark attacks, dangers 
of snake venom, and cauliflower ears, and in the latter study, the nutritional value 
of food from various fast-food chains, sports drinks, and the chemicals in fire- 
works (Gebre, 2018; Gebre & Polman, 2016; Polman & Gebre, 2015). 

The students received feedback on their progressive infographic drafts from 
their peers and from volunteer qualified science professionals. From the exam- 
ination of what students had learned about infographic creation through the 
feedback and the quality of the students’ final versions, issues for progressing 
the incorporation of infographics into multimodal disciplinary literacy were 
revealed. These included the propensity of students to use predominantly iconic 
images, such as photographs and drawings, rather than schematic images, such as 
diagrams, charts, and graphs, and their difficulties in strategically integrating the 
use of images and language. 

The inter-relating of language and images was referred to by Polman and 
Gebre as student management of “the interdependence of the visual and tex- 
tual descriptions” (2015, p. 885). They analysed the distribution of meaning 
across the modes of language and images (which they called non-text representa- 
tions). Non-text representations included iconic images, such as photographs or 
drawings, schematic images or diagrams, and charts or graphs. The message or 
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meaning communicated about the topic by each representation was referred to 
by the researchers as a dimension of meaning. A dimensionality ratio for each 
infographic was calculated by dividing the number of dimensions conveyed by 
non-text representations by the number of non-text representations in the same 
infographic. A non-text representation might communicate one or several dis- 
tinct dimensions of meaning, or zero dimensions if the information it conveys is 
already conveyed in the text or in other non-text representations. 

Gebre and Polman believed that their dimensionality index showed the 
degree to which learners communicate “more with less” and do not overuse 
non-value-adding representations (2016, p. 2678). The researchers did not 
take into account any dimension of meaning that was conveyed via text alone. 
This severely limits insight into the functional complementarity that may be 
obtained between text and image in some infographics. The co-occurrence of 
the same dimensions of meaning in text and image is considered in this study 
to be undesirable, but this apparent redundancy may be strategic. For example, 
Figure 8.5 shows an infographic response by a Year 11 physics student to the 
task of explaining the operation of a simplified direct current electric motor 
(Unsworth et al., 2022). The first two sentences may be considered redundant 
with the visual depiction of these meanings in the diagram. However, the inclu- 
sion of these sentences enables the use of the text reference “This” in sentence 
three, to clarify agency in the causal relationship between the current flow and 
the rotation of the coil, which can only be conveyed in language. Condensing 
meaning in infographics is a complex process, the investigation of which clearly 
needs to be based on a more detailed and subtle account of intermodal cohesion 
(Unsworth, 2020a, 2021). 

The approach taken by McDermott and Hand (2013, 2016) to investigating 
students’ co-articulation of image and language in their multimodal responses 
to science tasks was adopted in a number of related studies reported in Hand, 
McDermott and Prain (2016). McDermott and Hand (2013, 2016) designed 
a one lesson intervention with high school chemistry students to build their 
awareness of ways to improve the integration of non-text modes in their mul- 
timodal representations, which they referred to as ‘embeddedness’. Teachers 
were provided with a lesson plan outline, which drew attention to strategies for 
linking modes, such as “placing modes other than text near text that refers to 
them, complete textual descriptions of modes in the text, (and) captions added 
to modes other than text” (McDermott & Hand, 2013, p. 242). The lesson 
culminated with a joint student and teacher generated checklist for assessing 
the embeddedness of multimodal science representations. The researchers devel- 
oped a scale that provided an average embeddedness score (AES). Each use of 
a non-text mode was awarded a point for any of the following characteristics: 
next to the text, referred to in the text, scientific accuracy, including a caption, 
completeness (amount of detail), and originality (created by the student and not 
adopted from another source). The AES was calculated by adding the scores 
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FIGURE 8.5 Year 11 physics student infographic on the functioning of a direct cur- 


rent electric motor 


for each modal representation and then dividing by the total number of modal 
representations. 

The results of the studies provide evidence to support a strong positive rela- 
tionship between what the researchers refer to as “high levels of embeddedness” 
and student achievement on tests of science understanding (McDermott & 
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Hand, 2013, p. 223). But the AES rubric addresses other factors that are not 
germane to the students’ linking of different modes in their representations, 
such as originality (created by the student and not adopted from another 
source). The AES includes three inter-modal linking devices: next to the text, 
referred to in the text, and including a caption. A caption is frequently the 
titling or naming of an image, and it is not clear how this is a linking device. 
There is no mention of annotations to an image. Annotations may be naming 
labels, but they can also provide other information, such as indicating processes 
and cause-effect relationships among image elements. In referring to a modal 
representation being near text, it is not clear whether text refers to a ‘main’ 
text or to other forms of text segment typically embedded into infographic 
portrayals. There is also no mention of lines or arrows, which are frequently 
used to link image and text in these kinds of multimodal representations. From 
a semiotic perspective, the concept of intermodal linking addressed in research 
of this kind is confined to limited forms of textual meanings — physical link- 
ing through layout (‘next to text’) or reference (e.g. ‘see Figure 8.1’), and the 
presence of a caption. 

Some consideration of the co-articulation of image and language and the dis- 
tribution of meaning across modes was addressed in a study of year nine students 
learning about osmosis (Fuhrmann et al., 2018). The students conducted an 
experiment using an egg, vinegar, sugar, and water to show that the membrane 
of the egg is semi-permeable, allowing water molecules to penetrate but not 
sugar molecules. The students were asked to construct a diagram along with a 
written explanation showing what happened to the egg membrane and to the 
water and sugar molecules, labelling the egg membrane and individual mole- 
cules, and showing the direction that the labelled components would move. The 
rubric for the assessment of the students’ multimodal representations is indicated 
in the summary adaptation from the paper in Table 8.1. 

Although the study did not include any preparatory teaching of the co-artic- 
ulation of image and language, the communication Category D of the rubric in 
Table 8.1 certainly addresses this. Category D indicates that annotation of the 
drawing is an expectation, and further, that annotations are expected to describe 
the activity as well as to label entities. Acknowledgement of the complementary 
distribution of meaning across the modes of image and language is also partially 
evident, for example, in Category B where temporal progression can be repre- 
sented graphically or verbally. 

On the other hand, Category A suggests that scoring for the inclusion of 
micro-level representation can only occur via the image, whereas this might be 
included in language even if the image is represented at the macro level. One of 
the examples of student work from the Fuhrmann et al. (2018) study, includes a 
micro representation in the image as well as referring to molecules in an adjacent 
text segment. While the approach in this study deals more systematically with 
how image and language are co-articulated in multimodal representations, some 
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TABLE 8.1 The ‘model drawing rubric’ with the score and examples 


Category 


Score 


A: Macro—micro level 

Scores here indicate the extent to which the 
model represents both micro and macro 
levels of the experiment. 


B: Temporal chaining 
Score indicates whether students draw their 
model as a process or as a static state. 


C: Scientific explanation 

These scores measure the formal scientific 
understanding of the observed 
phenomenon: The variables involved 
in the osmosis process and their 
interactions. 


D: Communication 

These scores indicate the extent of the 
model’s design clarity and its capacity to 
communicate. 


(0) No picture. 

(1) Drawing contains a picture of an egg 
without further qualification. 

(2) Drawing contains some or all the 
kinds of particles and other elements 
identified for the model (sugar and 
water as distinct substances) with focus 
on detail at the molecular level. 

(0) No temporal chaining: Static state is 
indicated rather than a process. 

(1) Provides either a graphic 
representation of a two-step process 
or some other explanation of the 
temporal progression. 

(0) No explanation. 

(1) An incorrect explanation or an 
incomplete explanation that mentions 
only a single factor or uses concepts 
imprecisely. 

(2) A more elaborate, scientifically 
complete explanation includes several 
sentences and uses all chemical 
concepts correctly 

(0) No label or text in proximity to the 
drawing. 

(1) Student includes a word, words or 
arrows in the drawing, indicating an 
understanding of the value of labelling 
and illustrative graphic elements in 
visual representations of scientific 
explanations. 

(2) Student describes the movement 
of particles in a sentence or more 
and includes arrows and labels in 
drawings. 


commonly occurring realisations of inter-model cohesion are not incorporated 


such as the inclusion of reference to the image in the explanation (e.g. ‘see 


Figure X’), or through numbering or ‘lettering’ of image elements included in 


the explanation text. 


While many science education researchers recognise the increasing promi- 


nence of infographic communication in everyday life and its particular significance 
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in science and science education, the conceptualising of image—language inter- 
relationships in researching students’ multimodal representation construction in 
science learning remains fragmentary, inconsistent, or absent. For example, in 
response to the limited uptake of student-generated multimodal representations 
in science pedagogy, a framework for evaluating explanatory diagrams in chem- 
istry incorporated a rubric for classifying these student representations as non- 
explanations, macro-descriptions, mixed descriptions, associative explanations, 
simple scientific explanations, or complex scientific explanations (McLure et al., 
2021). The descriptions of these classifications do not mention language at all. 
However, examination of the examples for each classification reveals that the 
lower-order non-explanations and mixed explanations include no verbal annota- 
tion at all, the associative explanation has minimal annotation, but the simple and 
complex scientific explanations include many more annotations. This suggests 
that the role of language and its interaction with the image(s) in determining the 
quality of students’ infographic explanations warrants further investigation. 

A review of learning from and learning with diagrams in science education 
noted that they include a variety of visual, symbolic, and verbal features that 
collectively contribute to the representation of an idea or event (Tippett, 2016). 
However, there was no discussion in the review about how to describe or teach 
the co-articulation of these different modes that comprise this collective rep- 
resentation. The recommendations from the review were confined to explicit 
teaching of the scientific conventions such as captions, labels, symbols, and the 
use of arrows. Advancing 21st-century multimodal literacy to incorporate info- 
graphics as a nascent but increasingly significant genre in communicating scien- 
tific knowledge in the school curriculum and in the broader community will 
require transdisciplinary attention to the multimodal rhetoric of the genre to 
inform research and development of enhanced multimodal disciplinary literacy 


pedagogy. 


Mapping options for image-language integration and meaning 
aggregation in infographics 


Inspired by the work of Bateman (2008) and his colleagues (Bateman et al., 
2017; Hiippala, 2016, 2020; Hiippala et al., 2021), recent and ongoing semiotic 
research analysing infographics in high school science textbooks has started to 
document the repertoire of options for ways in which images and language are 
deployed in these multimodal representations (Martin et al., in press; Martin & 
Unsworth, forthcoming; Unsworth, 2020a, 2021). As shown in Figure 8.6, info- 
graphics are made of two categories/modes — images and language. These can 
then be categorised further. The two sub-categories of language are co-text and 
annotations. The left-facing brace means that any one or all of the options to 
the right of the brace may occur in the infographic. The dotted line indicates 
that other, as yet unspecified image options may be included. The dash after the 
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photograph" 


drawing" 


IMAGE diagram" 


graph” 


INFOGRAPHIC map" 


caption 
>| — tL, t block 
LANGUAGE _ sais line linked 


superimposed bracketed 
annotation >| { connected call out 
eL 
— aligned keyed 


FIGURE 8.6 Co-deployment options for image and language in infographics 


line brackets means that the single option within that bracket may or may not 
be present. 

The image components of infographics can consist of one or more of the 
several types of images, such as photographs, drawings, diagrams, and graphs, 
or a combination of the different image types. For example, the infographic 
in Figure 8.1 includes several connected diagrams, Figure 8.2 has two different 
drawings and Figure 8.3 has drawings, maps, and diagrams. 

The language components of infographics can include co-text and/or anno- 
tations. The co-text, which relates to the image as a whole, can consist of a 
caption and/or one or more interpolated text blocks. An example of a caption 
referring to the image as a whole can be seen in Figure 8.1, where the caption is 
located to the top and right of the diagram of mitosis. A prominent interpolated 
text block is superimposed on the drawing of the sky in Figure 8.2, highlighted 
in Figure 8.7. 

This text block also refers to the whole combination of the drawings of 
the land and the ocean. Annotations relate to specific parts of the image. The 
relationship of annotations to particular parts of the image can be specified in 
several ways. These include, for example, in Figure 8.1, by connecting lines 
linking annotations to the relevant parts of the cells, or by the close alignment 


Increased greenhouse gases from human activities 
result in climate change and ocean acidification. 


CLIMATE CHANGE = OCEAN CHANGE 


FIGURE 8.7 Text block within Figure 8.2 


Source: National Oceanic and Atmospheric Administration of the US Department of Commerce 
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of annotations with the relevant image elements as shown in Figure 8.2, where 
‘deforestation’, ‘fertilising crops’, ‘raising livestock’, etc. are aligned with the cor- 
responding elements in the drawing. There are also other connecting devices 
such as call-outs, brackets, and keys. 

The distinctive co-deployment of image and language resources in infographics 
serves to aggregate meanings in a multimodal construal of complex phenomena 
as a synoptic ‘eyeful’ within a single page format. To describe this condensation 
of meaning we need to relate the nature of the infographics as image—language 
ensembles to the ideational meanings that construe the represented phenomena 
in terms of the relevant field knowledge (An example of which was seen in 
Figure 8.6). Doran and Martin (2021) described the representation of phenom- 
ena in these multimodal assemblages in science texts from the perspective of field. 
They view field as a resource for construing phenomena as sequences of activities 
alongside the taxonomies of items involved in these sequences organised by both 
classification and composition, and their associated properties. 

In language, activity is typically realised by verbs. For example, in explaining 
sound waves we may say: 


A vibrating object compresses the air particles next to it and these com- 
pressed air particles compress the air particles next to them and so on, so 
the compression wave travels away from the vibrating object. 


Here, the activity sequence in the first sentence, which is expressed by the verb forms 
of “compress”, is condensed in the second sentence into the noun form “compres- 
sion wave”. Even though the “compression wave” is grammatically a noun group, 
it is not actually naming a ‘thing’, but an activity sequence that is represented in 
language as if it were a thing. In this way, nominalisation is one significant linguistic 
resource for condensing meaning. The use of nominalisation in highly condensed 
discourse is also associated with the expression of logical relations in noun or verb 
form rather than their more common expression as conjunctions. So, for example, 
relations most frequently realised by conjunctions, like ‘if? or ‘so’, may be realised 
by nouns like ‘condition’ and ‘result’ or verbal forms like “depends on’ or ‘leads 
to’. Balancing condensation of meaning with accessibility by lay or novice readers 
involves careful articulation of the highly condensed nominalised disciplinary dis- 
course with the verb and conjunction focussed, and more familiar, everyday/spoken 
grammatical forms. Language also has a range of specialised forms for realising part- 
whole taxonomic relations and classification taxonomies (Hao, 2020; Martin, 1992), 
but infographics more usually convey these kinds of field relations through images. 
In their seminal work on the grammar of visual design, Kress and van Leeuwen 
(2006, 2020) distinguished between images that represent activity (which they 
called “narrative structures”) and images that represent classificational taxono- 
mies, as well as those that represent compositional (part-whole) taxonomies, 
which they referred to as “analytical structures”. It is unclear in Kress and van 
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Leeuwen’s (2006, 2020) approach whether images can contain, and accord the 
same status to, more than one of these types of structures (Doran, 2019). Kress 
and van Leeuwen (2006) do indicate that timelines seem to occupy an interme- 
diate position between the narrative and the analytical, and that genealogies and 
evolutionary trees and graphs may blur the boundaries between the static and 
the dynamic (Kress & van Leeuwen, 2006, pp. 101-103). They also suggest that 
“embedding” can occur in diagrams and provide one example where a narrative 
structure is embedded within an analytical structure (p. 51). But certainly, the 
overwhelming emphasis in their work is on single structure images. However, 
Doran (2019) and Martin (2020) point out that infographic representations fre- 
quently simultaneously represent two or more of these aspects of field, and that 
the images combined with annotations and interpolated text blocks aggregate 
meaning multimodally in highly condensed synoptic portrayals. 

The nature and extent to which activity, classification, composition, and proper- 
ties are combined through images only, language only, or in combinations of image 
and language in an infographic, is referred to as aggregation. The system for aggre- 
gation in Figure 8.8 interfaces the options for inclusion of the technicality constru- 
ing field — composition, classification, activity, and property — with two options for 
combining those meanings in infographics — accumulation and integration. 

Accumulation refers to the aggregation of different types of meaning (com- 
position, classification, activity and property) from two or more ‘macro-groups’ 
within an infographic, each consisting of an image +/— annotations and +/— 
interpolated text blocks. Integration refers to aggregation within a macro-group 
through: (a) incorporation of two or more of the dimensions of activity, classifi- 
cation, composition, and property within one image; (b) representation of differ- 
ent dimensions of meaning construing field in the image and in the annotations; 
and (c) depiction of activity as a verbal representation of an entity and a visual 
representation of an action process. Aggregation in infographics may involve 
either or both of the options of accumulation and integration to a greater or 
lesser extent, hence in Figure 8.8, these options are represented as a cline. 

A highly condensed multimodal summative account of the biological func- 
tioning of inflammation is illustrated in the infographic from a senior high school 
biology text in Figure 8.9 (Greenwood et al., 2021). The prominent rhetorical 


{of accumulation 

AGGREGATION integration 
activity 
classification 
composition 
property 


FIGURE 8.8 Aggregating meaning in infographics 
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locus of this infographic is represented by the two salient diagrams. To the left, 
the diagram and annotations detail the infection, vasodilation, and phagocyte 
migration response. Connecting lines link most annotations to the relevant dia- 
gram segments, but bracketing is also used to indicate the tissue layers. Within 
this diagram, aggregation occurs through integration since the compositional 
relations among the entities are depicted visually and the activity of the inflam- 
mation process is communicated verbally through the annotations. Here, the 
language is overwhelmingly verb-focussed (although one nominalisation is given 
in parentheses — “vasodilation”). 

To the right, the second diagram is a ‘blow-up’ from the circled segment of 
the first diagram. Here again, aggregation is via integration as the composition 
is shown visually with labelling annotation and the activity of phagocytosis is 
communicated via the verb-focussed annotations. But aggregation also occurs 
via accumulation since it is the interaction of both diagrams that construes the 


The inflammatory response 


The inflammatory response (inflammation) is triggered by the presence of a wide range of harmful factors including the presence of 
infectious agents (pathogens), damaged cells, toxic compounds, foreign objects, or physical damage (e.g. burns). It is therefore a 
very common response and an important component of innate immunity. The purpose of the inflammatory response is to remove the 
dangerous stimuli and to stimulate the beginning of the healing process. Body parts or tissue affected by inflammation are usually 
showing signs of heat, pain, redness, swelling, and loss of function. 


P : i Red blood i 
Bacteria entering Chemicals (e.g. histamines Santer cells anny was 


on knife or other and prostaglandins) are 
sharp object. released by damaged cells, 
attracting more and more 
phagocytes to the infection. 


Blood clot 
forms 


Neutrophil 


sjwuepida 


Bacteria 
x 

‘ 
, 


Macrophage 


- 


enssy 
snoeuejnogng + siwieq 


M An abscess starts to form after 
Blood vessels MASS a few days. This collection of Phagocytes squeeze Bacteria are engulfed and 
diameter (vasodilation) z 
and permeability. dead phagocytes, damaged between cells making destroyed by phagocytes 

g tissue and various body fluids up blood vessel walls. (macrophages and neutrophils). 
is called pus. 


Stages in the inflammatory response 


Increased diameter and Phagocyte migration and Tissue repair 
permeability of blood vessels phagocytosis Functioning cells or supporting 
Blood vessels increase their Within one hour of injury, connective cells create new tissue 
diameter and permeability in the phagocytes appear on the scene. to replace dead or damaged cells. 


area of damage. This increases They squeeze between cells Some tissue regenerates easily 
blood flow to the area and allows of blood vessel walls to reach (skin) while others do not at all 
defensive substances to leak into the damaged area where they (cardiac muscle). 

tissue spaces. destroy invading microbes. 


FIGURE 8.9 Infographic: The inflammatory response 
Greenwood et al., 2021, p. 124 
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overall process of inflammation. The ultimate condensation of meaning occurs 
with the text blocks below the images. Here, what has been succinctly multi- 
modally detailed above, is synopsised verbally in the highly nominalised “Stages 
in the inflammatory response”. And the stages are condensed as the headings of 
each text box and connected by sequence arrows. 

As Martin (2020) points out, multimodal synopses in such infographics high- 
light the complex, compositional relations and the multi-tiered nature of the 
activity sequences that make up processes such as inflammation. High school sci- 
ence textbooks are replete with infographics depicting complex processes with 
two or more of the field dimensions of activity, classification composition, and 
property aggregated through integration by combining these dimensions multi- 
modally within a diagram and/or through accumulation, requiring the articula- 
tion of meanings across two or more diagrams (or other types of images). This 
is clearly shown in Figures 8.1, 8.2, and 8.3. The multimodal condensation of 
meaning with progressive nominalisation that we saw in the synopsising of the 
stages of inflammation in Figure 8.9 can also be seen in the ultimate synop- 
tic representation of the stages of mitosis (Prophase, Metaphase, Anaphase, and 
Telophase) in Figure 8.1. 

Martin (2020) aptly encapsulates the utility and the challenges of such info- 
graphics as resources for learning in schools: 


Where well supported by reading and classroom discussion they can work 
effectively as summative aggregations of accumulated knowledge; where 
supportive reading and classroom interaction has not taken place they may 
well function as impenetrable obstacles to teaching/learning. There is cer- 
tainly nothing transparent about the knowledge structure they encode. 
(Martin, 2020, p. 138) 


Mapping the inter-modal semiotic options drawn upon in infographics to suc- 
cinctly portray the construal of complex phenomena in a meaningfully accessible 
and appealing manner, can inform enhanced literacy pedagogies to support stu- 
dents in interpreting such texts and in constructing them to communicate what 
they have learned. 


Teaching-learning experiences to enhance infographic 
literacy development 


We believe it is important for students to develop a meta-semiotic understanding 
of infographics, along with other representational genres they need to engage 
with. This means having explicit knowledge of the meaning-making resources 
of language and images to enable interrogation and discussion of how mean- 
ings are represented and could have been represented differently. This meta- 
semiotic knowledge provides students with a conscious repertoire of multimodal 
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meaning-making strategies they can draw on in constructing multimodal repre- 
sentations such as infographics. Developing students’ meta-semiotic understand- 
ing relies on teachers’ knowledge of the ‘grammars’ of language and images as 
meaning-making resources, as well as an understanding of the options for their 
deployment in representational genres, such as infographics, as outlined in the 
approach to intermodal mapping discussed in the previous section. This kind of 
teacher pedagogic content knowledge makes it possible to develop with students 
a shared critical understanding of how meanings are made in multimodal repre- 
sentations (New London Group, 2000). 

It is also important that this kind of meta-semiotically informed multi- 
modal literacy development is integrated with, or infused into, the teaching 
and learning of the disciplinary knowledge in the relevant field — such as sci- 
ence (Unsworth et al., 2022). However, this does not occur incidentally and 
needs to be deliberatively integrated into pedagogic practice. Hence, we assume 
a pedagogic orientation which combines fostering student enquiry alongside 
teachers’ guidance — through interaction in the context of shared experience 
(Martin & Rose, 2005). An effective pedagogic model for this involves teaching/ 
learning cycles (Martin & Rose, 2008). These cycles unfold in a series of steps — 
building field knowledge, teacher-modelled deconstruction of sample texts, 
joint teacher-student text construction, and, ultimately, independent student 
construction (Rothery, 1994). Various reconceptualisations of the cycle have 
emphasised that it can be entered at different phases, with differing emphases on 
interpretation and production, and with phases being recycled according to the 
needs of the student group (Rose & Martin, 2012; Unsworth, 2001). 

Within this orientation to multimodal literacy pedagogy, we outline a ‘5Cs’ 
approach to developing students’ critical interpretation and effective creation of 
infographics in the context of scientific literacy education. 


1. Contextualising infographic interpretation/creation through direct and/or 
vicarious engagement with the materiality of the target phenomena, so 
that students’ conceptual understanding is progressed through learning tasks 
based on direct experience and/or video, simulation, and illustration. 

2. Comprehending meanings realised by unfamiliar visual representations and/or 
by linguistic forms of expression more characteristic of academic or disci- 
plinary discourse than everyday language. Teachers ‘talking out’ (Lemke, 
1989, 1990) the unfamiliar visual or verbal realisations of the different types 
of meanings can clarify the significance of all parts of the infographic — in 
dialogue with students. For example, the “pH levels” in Figure 8.2 and the 
somewhat subtle indication of the map ‘blow-up’ in Figure 8.3 may warrant 
such ‘talking out’ for some students. 

3. Connecting meanings across different images and/or across images and text 
blocks, etc. may involve convergent or complementary meaning rela- 
tions. For example, we noted the complementarity relationship between 
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the meanings conveyed via the diagrams and the corresponding col- 
umns in the table below them in Figure 8.1. Some students may read 
the macro-groups discretely without attending to how they may relate to 
each other. Students could be asked to identify commonalities and circle 
each one in a different colour (on reproductions of the page) to highlight 
this consolidation across macro-groups. They might then be asked how 
convergent the meanings are in each case, and how one of the realisations 
may elaborate, extend, or enhance the other. The next move might be 
to highlight the meanings in the diagrams that are not committed in the 
text blocks, etc. 

4. Comparing infographics dealing with the same topic can help identify dif- 
ferences in the meaning of commitment in corresponding infographics. For 
example, Figure 8.1 could be compared with a corresponding infographic 
on the same topic in a different textbook (e.g. Lofts, 2015, p. 28; Silvester, 
2016, p. 10). The centriole pairs in the original colour versions of the 
diagrams shown here in greyscale in Figure 8.1 are readily discernible due 
to the clarity of their depiction and their red colour contrasting with the 
blue of the cytoplasm. On the other hand, the centrioles in a correspond- 
ing infographic in a different textbook (i.e. Silvester, 2016) are extremely 
difficult to discern and do not seem to be present in the depiction of late 
prophase (when they are in fact present in this phase of mitosis). Further, in 
Figure 8.1, the small circle representing the centromere is clearly visible 
in joining the double strands of the chromosomes, whereas in the other 
textbook, the centromere is not distinctly discernible at all (Martin et al., 
in press). 

5. Constructing multimodal responses to explanatory challenges has been shown 
to enhance the engagement and learning outcomes of science students 
in primary and junior high schools (Hubber & Tytler, 2017; Tytler et al., 
2017, 2018, 2020). This approach has concentrated on students drawing 
to explore explanations of phenomena; in general, it has involved limited 
explicit teaching about and modelling of representational options and inter- 
modal relations. However, as students move from junior to senior high 
school, they frequently need to construct succinct infographic representa- 
tions in assessment contexts. Building this competence benefits from teacher 
modelling and joint construction of infographics with students to advance 
the effectiveness of students’ independent constructions. 


As well as the foregoing types of learning experiences, various other scaffolding 
learning experiences can be used. These include the teacher guiding students 
to reconstruct existing infographics based on a critique of the originals jointly 
developed by the teacher and the students, or similar teacher-guided student 
construction of composite infographics derived from others collected by the 
students. Teachers can integrate semiotic description as part of their pedagogic 
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guidance enhancing students’ metarepresentational competence in their inter- 
pretation and creation of infographics as well as their substantive subject area 
learning (Disessa, 2004; Kozma & Russell, 2005). 


Recommendations for researching infographics literacy for 
digital futures 


Research in infographics and scientific literacy development will benefit from 
advancing a transdisciplinary approach that integrates perspectives from relevant 
disciplines, including science, health, and physical education, as well as social 
semiotics, literacy, and digital technologies. With the current impetus toward 
infographic representation in public and school-based science-related education, 
further developing contemporary research seeking to enhance student compe- 
tence in interpreting and creating infographics is essential. This development 
needs to include semiotic perspectives on the rhetorical deployment of integrated 
image and language resources for condensing meaning into succinct multimodal 
page-based portrayals. There is also a need to extend insights from initial research 
into the positive impact of developing meta-semiotic competence to support 
student learning through multimodal infographic representations (Disessa, 2004; 
Kozma & Russell, 2005). 

Additionally, educational research needs to rapidly prioritise substantially 
greater engagement with the strongly developing trajectory of digital info- 
graphics. A cursory internet search reveals an impressive array of animated 
science infographics (e.g. Lutz, 2021 — https://eleanorlutz.com/animated-sci- 
ence-infographics), which need to be acknowledged by schools as 21st-century 
digitally oriented students increasingly take to the internet as their personal 
learning resource. As students develop computer programming competencies 
from their early school years, they are using coding software such as Scratch 
(Massachusetts Institute of Technology, 2022) to code animated represen- 
tations of their science learning (e.g. Goletti, 2018; Ko, 2019; Nikmah & 
Ellianawati, 2019; Robertson et al., 2021). It is imperative that research into 
infographic literacies for digital futures assumes transdisciplinary perspectives 
so that literacy pedagogy is informed by techno-social—semiotic-—scientific 
epistemologies (Unsworth, 2020b). 
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ADVANCING ANIMATED STORY 
COMPOSITION THROUGH CODING 


Viewing animated stories remains hugely popular with children and continues to 
attract a massive adult audience. The opportunity for children to create animated 
stories greatly increased with the advent of child-accessible digital animation 
software (Chandler, 2013; Chandler et al., 2010). Today, animation software for 
children is readily available, and some 2D and 3D applications, such as Synfig, 
Opentoonz, Tupi, and Blender, can be accessed gratis online. A significant futures- 
oriented development in animation authoring opportunities for children and 
teenagers (and older animators) has been the emergence of the ‘programming as 
writing movement (Breen, 2016; Burke et al., 2016; Hagge, 2021; Hassenfeld & 
Bers, 2020). This occurred with recognition of the importance of introducing 
computer programming concepts to young students as a fundamental compe- 
tency, increasingly required in many dimensions of life in the computer-based, 
digitally connected, vocational, social, personal, and political environments of 
the 21st century. 

The concept of programming as writing grew from the creation of object- 
based or block-based approaches to computer programming or coding, such as 
the Alice suite, culminating in Storytelling Alice (Kelleher et al., 2007; Pausch & 
Forlines, 2000), and the development of Scratch (Resnick et al., 2009). In Scratch, 
blocks are puzzle-piece shapes in software that are used to create computer pro- 
gramming code. The blocks connect to each other vertically like a jigsaw puz- 
zle. There are ten categories of blocks (Motion, Looks, Sound, Event, Control, 
Sensing, Operators, Variables, List, and My Blocks), each with its own colour. 
The blocks enable users to program their selections or creations of characters 
or objects (called sprites in Scratch) and backgrounds, to generate the event 
sequences, dialogue, sound, and music that may comprise an animation. 
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From very early initiatives in teaching school children to code with Scratch, 
creating animated stories has been the main focus of these innovative programs 
(Burke & Kafai, 2010). These initiatives were predominantly conducted outside 
of children’s regular school classroom experience, and this has overwhelmingly 
remained the case for research on coding animated narratives to date. Since 
several countries have mandated the introduction of coding in school curricula, 
it is increasingly incorporated into science, technology, engineering, and maths 
(STEM), but relatively little uptake has occurred in humanities and English 
Language Arts (ELA) and other first language curricula (Rich et al., 2018; Whyte 
et al., 2019). However, outside of school, a growing number of child coding 
enthusiasts in online communities have generated a plethora of stories ranging 
from the brief and unimaginative, to intriguing and insightful micro-narratives, 
and extended and episodic animated movies (Burke et al., 2016; Hagge, 2018; 
Resnick & Rusk, 2020). 


Refocussing programming as writing for integration in 
language arts curricula 


If digital animation as a prominent 21st-century literate practice is to be acces- 
sible to all students both as consumers and creators, it is important that language 
and literacy curricula introduce the expanded communicative potential of ani- 
mation that coding affords. This is the aspiration of the ‘programming as writing’ 
and the ‘coding as literacy’ research agendas (Breen, 2016; Burke et al., 2016; 
Hagge, 2021; Hassenfeld & Bers, 2020), but for these agendas to achieve traction 
in schools, language arts teachers need to be given opportunities to develop their 
own competencies in coding. It is also essential that research investigating the 
melding of the teaching of coding and animated story creation is able to indi- 
cate how pedagogies for coding animated narratives can simultaneously address 
language arts curriculum goals for developing students’ narrative interpretation 
and composition. 

By far, the most popular block coding software for school-age students is 
Scratch, with over 21 million registered users predominantly between eight and 
sixteen years of age (Hagge, 2018; Resnick & Rusk, 2020). It is therefore not 
surprising that the vast majority of studies examining pedagogy development for 
coding animated stories have used Scratch. A key issue is how the trajectory of 
these studies is oriented toward enabling the integration of coding and animated 
story creation as part of the language arts curriculum. Research from 2010 to 
the present using Scratch for story creation with students from eight to fourteen 
years of age has occurred almost entirely outside of regular classroom teaching in 
after school clubs or elective programs (Burke, 2012; Burke & Kafai, 2010, 2012; 
Whyte et al., 2019, 2020). Researchers with expertise in Scratch conducted these 
programs with groups of about ten to twelve students once or twice a week over 
periods of six to seven weeks. 
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A detailed account of the highly scaffolded pedagogy was provided in the 
Burke and Kafai studies (Burke & Kafai, 2010, 2012) based on a well-known 
approach to ELA narrative composition pedagogy (Calkins, 1986). The main 
instructional stages are: pre-writing; drafting; revising; editing; and publishing. 
Each of the Scratch sessions opened with a mini lesson introducing an element of 
effective composition (such as characterisation, foreshadowing, setting a scene) 
tied to a coding procedure in Scratch (e.g. broadcast feature to establish dialogue, 
importing external images, using loops), and every mini lesson included one to 
three sample Scratch stories, exemplifying the storytelling element or genre (e.g. 
mystery, action/adventure) featured within the lesson. The sample stories also 
provided the opportunity to examine the coding scripts to understand how the 
targeted story elements were coded (Burke & Kafai, 2012). 

These researchers emphasised the students’ use of storyboards in preparing the 
stories as well as in drafting and revising. They fostered student collaboration by 
including gallery walks enabling students to share drafts and feedback provided 
by the instructor on draft stories. A highly scaffolded approach was also taken in 
the Whyte et al. studies (Whyte et al., 2019, 2020), albeit with less detail about 
the pedagogic procedures. They involved the students in a series of preparatory 
tasks that linked elements of story to the coding procedures required to produce 
them as Scratch animations, prior to the students coding a story independently. 
The second of these studies refined the pedagogy, emphasising the nature of 
narrative structuring and including more demonstrations of worked examples of 
preparatory tasks and of completed story structures. 

Some regular classroom programs involving coding Scratch stories have been 
reported as brief descriptive outlines. For example, Hagge (2017) outlined a Year 
6 program where students created a Scratch story to illustrate their understand- 
ing of important events and character responses in a novel they were studying. 
To prepare the students to create their story, the teacher presented examples of 
the kinds of digital stories he wanted them to create. In discussing each example 
with the students, they observed how the available tools in Scratch were used and 
examined the coding scripts to understand how the use of coding tools aligned 
with the design and content of each story. The teacher ‘conferenced’ with each 
student during the design and coding process to assess their progress in coding 
the story and to provide assistance, as required. The students presented their 
completed stories in class, discussing how character response was communicated, 
and later published their stories on the Scratch website. Greater specification of 
the pedagogy including references for the teacher’s sample stories and some 
examples of the stories created by the students would better enable teachers to 
consider adopting the approach introduced in the study. 

Pektaş and Sullivan (2021) interviewed two fourth-grade students who par- 
ticipated in a Scratch program in their regular ELA classroom. In this six-lesson 
unit of work, students were taught to identify character traits and behaviours 
associated with them, identify their own character traits, and create a Scratch 
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story based on a few character traits. No further information is provided about 
the classroom program. The interviewed students were members of a group who 
composed a story called Zombie Apocalypse in which Earth is hit by a meteor con- 
taining zombies who then destroy the planet and zombify survivors. The inter- 
view deals only with the introduction to the story that the two interviewees were 
responsible for producing. This extends as far as the meteor hitting Earth; the 
planet being destroyed and the zombies about to spread all over the world. One 
part of the interview data deals with one student’s coding of a sequence of blocks 
to simulate the glide block in Scratch to portray the meteor moving towards Earth 
as well as the background changes as the planet is destroyed and zombies appear. 

The other aspect of the interview data related to the selection and modi- 
fication of sprites, which is an important aspect of Scratch story creation. This 
interview data discussed the remixing of sprites by the other interviewee to 
create a character with protective clothing and also black skin similar to the 
interviewee. However, there is no account of this character’s role in the story 
and no further discussion of the story as a whole or how it was coded. In fact, 
all of the studies mentioned would have benefitted from greater specification to 
enable ELA teachers to see how this multimodal composition context facilitates 
advancing students’ understanding and implementation of narrative composition 
techniques such as rhetorical structuring, characterisation, point of view, etc. 

Only one complete student Scratch story is accessible from the studies by 
Burke and his associates (Burke, 2012; Burke et al., 2016; Burke & Kafai, 2010, 
2012). This is the Crayfish story created by one group of students aged 12-14 
years (Burke & Kafai, 2012). An image of the six scenes of the story is included 
but no reference is provided to the story on the Scratch website. [None of the 
other stories referred to or partially described are accessible on the Scratch web- 
site.] In the Crayfish story, the single boy character is given a crayfish, who takes it 
home, puts water in the crayfish bowl, the crayfish exclaims in panic at the addi- 
tion of fresh water, asks where the salt is, and dies. All scenes are distant views 
of the characters. The extremely basic orientation, complication, evaluation, 
resolution structure, lack of characterisation, simplistic point of view, etc. repre- 
sent only the most minimal narrative compositional competence for a pre/early 
teenage student, yet the authors indicate that this 90-second story took the stu- 
dent more than ten hours to code. These studies report introductory programs 
for largely novice student coders and, of course, greater coding efficiency would 
be expected as students gain more experience, but we indicate in the subsequent 
sections of this chapter that there needs to be concomitant development in the 
quality of the students’ multimodal authoring. 

It is very clear from the published literature, and it is possible to determine from 
the website, that students of this age can certainly create sophisticated micro or 
extended animated narratives using Scratch (Resnick & Rusk, 2020). Researchers 
continue to pursue the explicit goal of the Burke and Kafai work to “leverage the 
professional knowledge of K-12 educators in traditional English/language arts 
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classrooms to better integrate programming into core-content classroom activi- 
ties” (Burke & Kafai, 2012, p. 433). There remains a need, however, for research 
into this integration to show more clearly how literacy curriculum authoring 
outcomes are advanced in the reported studies. Recent research using Scratch to 
investigate the integration of coding into the literacy curriculum through the 
creation of digital stories indicates that several students created stories with a 
complete narrative structure, but no examples of the stories are provided either 
in the research reports or by reference to the Scratch website (Whyte et al., 2019, 
2020). In the following sections, we discuss key issues for enhancing the integra- 
tion of coding and multimodal authoring pedagogy. 


Re-balancing programming as writing and multimodal 
authoring 


Young Scratch enthusiasts, quite independent of any schooling connection, 
have created sophisticated stories, frequently in online collaboration with other 
Scratchers. In at least one project, they have substantially advanced their coding and 
story creation competence with the impetus of mentor feedback (Fields et al., 
2014). While it is not possible to identify the age of Scratchers, an example of a 
readily accessible, highly sophisticated series of stories created by a young Scratcher 
named Taryn in collaboration with online peers is the Colour Divide (https:// 
scratch.mit.edu/studios/1393192/). According to Resnick and Rusk (2020), 
Taryn was introduced to Scratch at her school in South Africa at age ten. At some 
later date, she collaborated with online Scratchers to create the Colour Divide 
series, which explores social issues in South Africa and the scars left by apartheid. 

The Colour Divide stories are an exceptional animated narrative series, but we 
have also been able to identify quite sophisticated micro-narratives on the Scratch 
website and deduce the approximate age of the Scratcher. One example is Can I 
Come In? (https://scratch.mit.edu/projects/23367579) by FunnyAnimatorjimTV 
created in 2014. In 2016 FunnyAnimatorJimTV posted a youtube video on 
drawing in Scratch (https://www.youtube.com/watch?v=kmQvNoqIGbw) in 
which his speaking clearly indicates a boy’s voice, which has not yet broken with 
the advent of adolescence, so he was pre-adolescent when he authored Can I 
Come In? This animation demonstrates the kind of narrative technique that ELA 
teachers would consider reflective of appropriate development of compositional 
competence for this age level. While the story is just under 90 seconds, and the 
external event sequence is quite simple, the engaging impact of the story lies in 
the subtle and sophisticated characterisation that foregrounds the interiority of 
the two characters through their verbal and bodily responses. 

The story begins with Matt knocking on George’s door to which George 
responds that he will be “out in a minute”. An on-screen audience message then 
indicates that six minutes and twelve seconds have passed, and Matt is still wait- 
ing outside. He asks if he and George are going to have a play date as arranged, 
to which George replies that this can go ahead as soon as his television program 
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is finished. Matt inquires how long that will be, and George indicates a few min- 
utes. The next on-screen audience message indicates that eight and a half hours 
have passed. George, at last, opens his door into a night scene and sees a note on 
the floor from Matt. Matt’s note reminds George that George was the one who 
wanted the play date. The note expresses exasperation about why George won’t 
tire of watching television, while indicating that Matt had to go home. George 
muses about whether Matt may want to come over the next day for his sushi 
construction tutorial. 

The narrative techniques in Can I Come In? draw attention to potential shifts 
in research and pedagogy that may facilitate more widespread and effective 
development of coding as multimodal authoring within schools. The effective 
minimalist ‘stick figure’ drawings of the story characters and similarly simple set- 
tings enable the story to communicate the essence of the message. This counters 
any potential preoccupation with avatar choice or selection of elaborate settings 
that are peripheral to the story themes. At the same time, the story emphasises 
the importance of communicating attitudinal meanings of positive or negative 
affect — un/happiness, dis/satisfaction, and in/security — in establishing the inner 
life of characters, as well as portraying their actions in external reality. The range 
of different attitudinal expressions by the Matt character is shown in Figure 9.1. 

Studies investigating the integration of teaching coding and story creation 
using Scratch do not mention the portrayal of characters’ attitudinal responses 
(Burke et al., 2016; Burke & Kafai, 2010, 2012; Hagge, 2017; Pektas & Sullivan, 
2021; Whyte et al., 2019, 2020). These studies attend to story structure, such as 
motivating events, attempts to resolve, consequences and resolution of the nar- 
rative, and a concern with characterisation, though there is no mention of the 
characters’ evaluation of events, or the prosodic nature of this evaluation realised 
by characters’ attitudinal responses as the plot unfolds. This aspect of narrative 
technique is commonly required knowledge for students in senior elementary 
and junior high school in countries such as the US in the Common Core 
Standards for English Language Arts (National Governors Association Center 


FIGURE 9.1 Matt’s attitudinal expressions 
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for Best Practices, 2010, p. 36, 44) and in the Australian Curriculum for 
English (ACARA, 2018, content descriptions ACELT 1616, ACELA 1518). 
Such requirements can be readily addressed using Scratch and warrant explicit 
attention in research and pedagogy seeking to advance coding of animated 
narratives within ELA. 

The Can I Come In? story also highlights the multimodal authoring compe- 
tencies that complement coding capacity in creating an engaging narrative. The 
effective selection and synchronisation of sound and music contributes to the 
effectiveness of this story. For example, the music accompanying the first five sec- 
onds on-screen audience message indicating the passage of six minutes and twelve 
seconds in the story, is quite a lively melody. On the other hand, the accompany- 
ing music for the nine-second on-screen message indicating the passing of eight 
and a half hours is very much slower and lower in tone. The story also incorpo- 
rates background sounds of George’s television show, and night sounds of frogs as 
George finally emerges to see Matt’s note. As indicated in Figure 9.1, the author 
knows how to use depictions of facial expression and gesture to communicate 
attitude, but there is also evidence of his capacity to use focalisation to position 
viewers as if they were characters in the story. This occurs as George expresses 
surprise at seeing the note on the floor, and then the viewer is positioned as 
George reading the note (Figure 9.2). 

Multimodal authoring competencies including knowledge of the semiotics 
of body movement, gesture, posture, facial expression, interpersonal position- 
ing, sound, music, voice quality, colour, and knowledge of story structure options 
as well as the communication of engaging, personally, and socially significant 
experience — are all crucial to composing effective digital animated narratives. 
However, it is coding, in conjunction with the astute selection, modification, 
and/or composition of particular story-apposite sprites, that has greatly extended 


FIGURE 9.2 Viewer as character 
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authorial freedom to enable students to create their own entirely original digital 
animated stories. Clicking on the Scratch ‘See inside’ tab for the Can I Come In? 
story (https://scratch.mit.edu/projects/23367579) enabling an inspection of the 
block coding that constructs the story will readily indicate the advanced coding 
capacity of this author. 

The management of sprites in this story is very sophisticated. Rather than the 
usual animating of an entire character, this author separates the coding of facial 
expression and arm movement. The character’s mouth and arms are isolated 
as discrete entities separated from the body, as can be seen in the repertoire of 
sprites at the bottom right of Figure 9.3. Each of these sprites has multiple ‘cos- 
tumes’ (versions). The mouth sprite in row one, shown enlarged in the centre of 
the Figure, has 23 different costumes, some of which are visible in the column 
on the left in Figure 9.3. Selections from these costumes can be coded to appear 
in rapid succession, which results in visually depicted changes in the character’s 
facial expression. Other ‘mouth costumes’ can be selected and coded to appear 
in rapid succession, resulting in a visual simulation of talking approximating lip- 
synch with the spoken dialogue. Similarly, multiple costumes of the arm sprite 
can be coded to appear successively to produce the visual depiction of fluidic 
movement of the character’s arm in actions, such as knocking on the door. 

Representing changes in facial expression, posture, and gesture in the Can I 
Come In? story by creating discrete sprites with multiple costumes for the mouth 
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FIGURE 9.3 Separating coding of facial expression and arm movement from animat- 
ing the entire character 
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and arms and coding them to appear on the base representation of the character’s 
face required quite sophisticated coding competence, which may take most nov- 
ice Scratchers some time to acquire. However, novice Scratchers could closely 
approximate the representation of these changes in facial expression, gesture, and 
posture by creating multiple costumes for the character as a whole entity and 
coding the story to change selections from these whole character costumes in 
rapid succession. Each of the panels in Figure 9.1 could be separate costumes for 
this character. Coding the costume in the second panel to appear rapidly follow- 
ing that in panel one would result in the visual depiction of a fluid change in the 
character’s expression. In this way, changes in facial expression and gesture could 
appear in a fairly similar manner to those in the Can I Come In? story. Learning to 
code this costume sequencing and making it appear at the appropriate time in the 
story is well within the scope of early coding development for novice Scratchers. 

Many other techniques for engaging multimodal storytelling can be utilised 
that require only very basic coding to implement. For example, a close-up view 
of a character can be achieved by creating a copy of the sprite as a separate cos- 
tume, enlarging, and then cropping to show the enlarged head and shoulders 
only. This can be coded to follow a mid or distant view of the character in 
the story, which would require a simultaneous change in the background to be 
consistent with the close-up, all of which can be managed with basic coding. 
A simpler example enlarging the entire sprite is shown in Figure 9.4 from a story 
created by 13-year-old students in a regular English classroom, who had been 
learning to code stories for about seven weeks. The rabbit is a Scratch sprite, but 
none of its costumes includes a mouth. The students modified the sprite to cre- 
ate an additional costume with a mouth configuration consistent with the shout 
of surprise. 

Positioning the viewer as a character in the story is an effective way of engag- 
ing the audience and fostering empathy for the character, as occurs at the end of 
Can I Come In? when George is reading the note (Figure 9.2). This requires only 
the single sprite of the note with George’s (line) arms extending from it, and a 
small amount of basic coding. 


m 
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FIGURE 9.4 Modifying a sprite and using a close-up view 
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FIGURE 9.5 How one student changed the point of view 


Figure 9.5 indicates how another 13-year-old student, in the same class that 
had been learning to code for about seven weeks, incorporated this kind of shift 
in point of view. In this case, the background in the second frame should have 
been changed to be the same as the floor in the first frame, to make the shift in 
point of view appear more contextualised within the story world. 

Similarly, other multimodal authoring techniques, such as the incorpora- 
tion of sound, music, and spoken dialogue, are easily managed from a coding 
perspective. Re-balancing attention to coding and developing students’ digital 
multimodal narrative authoring is an important consideration in investigating 
approaches to integrating coding and literacy in ELA. 


Challenges in integrating coding animated narratives as 
multimodal authoring in ELA 


Research seeking to further the integration of computing and literacy through 
teaching students to code animated stories remains very much in its infancy. 
Progressing this agenda faces a range of challenges, including extending literacy 
teachers’ coding competencies and their knowledge of the multimodal semiot- 
ics underlying animation creation, as well as assessment regimes in language arts 
and literacy curricula that continue to privilege the composition of monomodal 
texts. Added to these are the pedagogic challenges of catering for the diversity 
of learning needs among students in regular classrooms. While some of these 
challenges are gradually being addressed, some may be accepted as enduring and 
others seen as insurmountable, one highly tractable challenge is in the hands of 
researchers. Investigations designed to promote the integration of coding and 
literacy pedagogy need to be able to show how this will facilitate and enhance 
the kinds of multimodal literacy outcomes expected in ELA curricula. 

The available literature investigating the integration of coding and literacy 
in ELA classrooms indicates that more attention to multimodal literacy out- 
comes is needed. In the studies to date, most students could learn enough basic 
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coding to generate a very simple story over about seven weeks, while fewer 
mastered more advanced coding, which might have been more efficient and 
reduced coding time. An important issue is whether and how those very simple 
stories, within the constraints of a novice’s basic coding competencies, might 
be enhanced to approximate the narrative qualities of some of the stories on 
the Scratch website created by enthusiast peers, which are more aligned with 
curriculum expectations for narrative composition. This would provide a basis 
for pursuing sustained cross-curricular collaboration between language arts and 
digital technology teachers for further integration of coding and literacy, and the 
inter-related learning development for students. 

Our examination of animated narratives of literary quality created by school- 
age Scratch enthusiasts, and those of novice coders in a regular ELA classroom, 
indicates the need to prioritise the contextualising of coding initiatives in ELA 
within the development of students’ interpretation and composition of multi- 
modal literary narrative. ELA curricula for students in the transition years across 
the upper elementary/primary and junior secondary school include a focus on 
students learning about narrative techniques. This involves the representation 
characters’ attitudinal responses to events in stories and the construction of point 
of view as key factors in audience (non)alignment with characters, which influ- 
ences audience engagement with the story, and its thematic interpretation. 

The Australian English curriculum for Year 6 requires students to explore 
“theme, characterisation, text structure, plot development, tone, vocabulary, sense 
of voice, narrative point of view, favoured grammatical structures and visual tech- 
niques...” (Australian Curriculum English Literacy (ACELT) content description 
1616) and to identify narrative strategies that “offer insights into characters’ feel- 
ings, so building empathy with their points of view...” (Australian Curriculum 
English Language (ACELA) content description 1518) (ACARA, 2018). 
Communicating evaluative meanings of affect, ethics, character, and capacity has 
been shown to be indexical of high-quality narrative writing by elementary and 
high school students (Macken-Horarik, 2003; Macken-Horarik & Sandiford, 
2016; Ngo, 2016; Rothery & Stenglin, 2000). The representation of this interi- 
ority of characters does not appear in the studies of students coding stories with 
Scratch, yet it could be readily incorporated in even the simplest of stories. 

In the Crayfish story discussed earlier, the same distant view of the one boy 
sprite is used throughout. An alternative representation might have been a close-up 
of the boy’s face with a smile emphasising his joy at receiving the crayfish. His 
subsequent appearance could have shown a changed facial expression when he 
realised the crayfish needed water. At the end of the story, the boy could be shown 
with a sad face and the crayfish dead at the bottom of the bowl. Empathy for the 
boy could have been increased if the depiction of the boy arriving home with 
the crayfish, showed only the boy’s arms extending out from the screen holding 
the crayfish tank directly in front, in a manner similar to Figure 9.2 — showing 
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George holding the note from Matt. This would have positioned viewers to have 
the point of view of the boy. 

While students and teachers are familiar with the everyday expression of atti- 
tudinal meanings through frequently used language, facial expression, and per- 
haps also through gesture, posture and movement, it cannot be assumed that they 
routinely draw on systematic knowledge of these meaning-making resources in 
interpreting and creating multimodal texts. Similarly, while teachers and students 
have experienced the use of close up, social, and remote depictions of characters 
as well as high and low angle vertical views, it cannot be assumed that they are 
attentive to the effects of these forms of representations in their interpretation 
or creation of multimodal narratives. Teachers and students need to be familiar 
with the ways in which images position viewers to have different points of view 
including — (i) as observers of story events, (ii) as if they were characters in the 
story, or (iii) as viewing events along with the viewpoint of particular characters 
(O’Brien, 2014). 

Research has provided systematic accounts of nuanced linguistic resources 
for communicating different kinds of attitudinal meanings (Droga & Humphrey, 
2003; Martin & White, 2005) and how such meanings can be conveyed in 
images (Economou, 2012, 2013; Martin, 2008; Tian, 2011; Unsworth, 2015). 
Research has also shown how to depict interaction and point of view in picto- 
rial narratives (Kress & van Leeuwen, 2020; Painter et al., 2013; Unsworth, 
2013a, 2013b). Studies conducted in primary schools have shown that when 
teachers are alerted to accessible and succinct descriptions of these meaning- 
making resources they can very effectively enable their students to successfully 
deploy these visual and verbal resources for constructing characters’ emotional 
responses, and to incorporate variation in point of view in the students’ creation 
of digital comics and iPad animations (Mills et al., 2020; Mills & Unsworth, 
2018; Unsworth & Mills, 2020). 

As well as contextualising the creation of Scratch stories in relation to curricular 
expectations for narrative composition, integrating coding and literacy in ELA 
needs to take into account the relatively recent advent of multimodal literacy in 
the curriculum in many countries such as in the Common Core State Standards 
for English Language Arts & Literacy in the US (National Governors Association 
Center for Best Practices, 2010), and English curricula in Australia (ACARA, 
2010) and (Singapore Ministry of Education, Singapore, 2010). However, these 
curricula tend to emphasise the interpretation of multimodal texts, rather than 
the creation of such texts. While multimodal digital compositions such as comic 
creation and the use of animation software for children have occurred in some 
classrooms (Chandler, 2013; Chandler et al., 2010, 2012; Unsworth & Thomas, 
2014), many teachers’ experience of story creation is monomodal, and the inclu- 
sion of images in student-created stories tends to be largely eliminated after the 
early years of schooling. 
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In many regular ELA classrooms, Scratch story creation initiatives will need to 
include support for teachers to provide explicit teaching of cinematic techniques 
and story creation as screenwriting competencies that have been successfully 
implemented in classrooms using digital comic maker and animation software 
(Mills et al., 2020; Unsworth & Mills, 2020). Learning to create animated mov- 
ies coordinating visual storytelling, scripting, and the incorporation of music and 
sound through coding, is a challenging form of multimodal composition that 
will require detailed pedagogic scaffolding for the majority of students. 

The importance of such well-developed pedagogic scaffolding for novice 
coders to create their animated stories is clearly reflected in existing studies. As 
previously indicated, the studies by Burke and colleagues (2010, 2012) included 
an examination of the coding of model stories, mini lessons on aspects of story 
creation, use of storyboards, student collaboration, and instructor feedback. 
Similarly, the approach by Whyte et al. (2019, 2020) was based on a contin- 
uum of preparatory learning activities conducted prior to students’ independent 
story creation. These learning activities ranged from teacher ‘directed’ to stu- 
dent ‘explored’ and included teacher demonstration of worked examples and 
completed story structures. However, these researchers have recognised that the 
skewing of the scaffolding towards building confidence and competence in cod- 
ing, while supporting rudimentary aspects of multimodal narrative authoring, is 
not sustainable if coding and literacy are to be integrated into ELA education. 
Burke et al. indicated that “there is also very much the need to explore (and 
make more explicit) the intersection between coding and narrative composi- 


, 


tion...” since their projects “largely held storytelling and coding apart as sepa- 
rate entities — each integral to and serving the same end goal — but introduced 
separately, nonetheless” (2012, p. 438). Correspondingly, Whyte and colleagues 
have indicated further development of their initiatives as necessarily working 
with teachers to determine how their perceptions of “...programming as a cur- 
riculum activity, and the role of multimodality in the literacy classroom could 


influence the co-design and implementation of these activities” (2020, p. 1323). 


Implications for a pedagogy of coding animated narratives 
in ELA 


From the perspective of integrating coding and literacy in ELA, contextualisa- 
tion of pedagogic practices within students’ developing knowledge of narrative 
technique is essential. While some Scratch stories by experienced student coders, 
such as The Colour Divide have episodes of about 15-minutes duration, most 
Scratch stories are about 90 seconds or less. Composing Scratch animations of 
this short duration is important to maintain the amount of coding required at a 
manageable level for novice coders to complete a story within a feasible time- 
frame. An important initial contextualising strategy, therefore, is raising both 
teachers’ and students’ awareness and appreciation of the micro-narrative genre, 
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and particularly the extensive popular cultural engagement with ultra-short, ani- 
mated movies. 

It is motivationally important to alert students to the significance of events, 
such as the annual Academy Award, or Oscar, for the Best Animated Short Film 
that began in 1932, and the plethora of Australian award-winning short animated 
movies can be viewed on the National Film and Sound Archive website (https:// 
www.nfsa.gov.au/collection/curated/australian-award-winning-animated- 
short-films). Many high-quality animated movies of less than three minutes’ dura- 
tion can also be readily accessed via YouTube, such as the moving Ticket without 
a Seat (https://www.youtube.com/watch?v=_o2kL_kbosg) or the highly amus- 
ing and moving Joy Story (https://www.youtube.com/watch?v=vT7FsWg1t28). 
Using resources such as these micro-narrative animations, groups of students 
within a class can be invited to identify the most enjoyable story they can find. 
These can then be shared and, through discussion, teachers can guide collabora- 
tive analyses to reveal their story design. Stories can be analysed to understand 
how they are constructed to captivate audience interest through the multimodal 
representation of events, as well as characters’ feelings. These are emphasised 
through filmic techniques, such as close-up views of characters, and sometimes 
by strategically positioning viewers as if they are viewing the story events as, or 
along with, a story character. 

In developing their knowledge of narrative authoring technique, a key cur- 
ricular focus for students in the transition years from primary to junior high 
school is the portrayal of characters’ evaluative responses to story events from 
different points of view. Incorporating this in pedagogy for coding animated 
stories is necessary to establish the credibility of coding as a viable and advanta- 
geous digital literacy resource in ELA education for students of this age. A clear 
corollary in introducing Scratch is giving more attention to the role of sprite 
selection, modification, remixing, importation, and/or creation. However, it 
is also important to ensure that students focus on the features of sprites that 
are integral to communicating the significant ideas in the story. This can be 
emphasised by drawing attention to the effectiveness of minimalist depictions of 
characters, such as in the Can I Come In? story. Students can view well-known 
animations which also use minimalist depictions, such as the popular contem- 
porary Australian television series, Bluey (https://iview.abc.net.au/show/bluey), 
or movies such as the Marjane Satrapi animated adaptation of her graphic novel 
Persepolis (Satrapi, 2008; Satrapi & Paronnaud, 2008) (https://www.youtube. 
com/watch?v=3PXHeKuBzPY). Students’ critical viewing of animated narra- 
tives is an important resource in their learning to create their own animated 
stories. 

The integration of coding and literacy in ELA that is accessible to all students 
will benefit from establishing a repertoire of pedagogic practices for developing 
low-, mid-, and high-level coding as authoring competencies. Such a reper- 
toire would facilitate the kind of sustained long-term learning progression that 
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would support students in the early and emergent stages of coding experience 
and extend those who are more advanced. For early stages, there is a plethora 
of tutorial activities on the Scratch website, but also elsewhere online such as on 
YouTube. For students who need additional support, these can be supplemented 
by learning activities gradually being accumulated through teacher experience. 

In a recent study, a teacher introducing Scratch to bi-lingual Spanish-American 
sixth-grade students asked them to compare a segment from a familiar telenovela 
(soap opera) to a Scratch story version (Ascenzi-Moreno et al., 2020). First, the 
students acted out the scene using a playscript and then watched a Scratch anima- 
tion depicting the same scene. They were then asked to match the script used 
by the telenovela actors with the Scratch code blocks. The first step was using 
the colours of the coding blocks for different functions (e.g. speaking, moving) 
to colour code the script according to the type of action. The second step was 
to draw a line from the relevant section of the script to the corresponding code 
blocks. Some, such as speaking (purple blocks) and sounds (pink blocks), were 
easy to connect, but some of the events (yellow) and motions (blue) were more 
challenging. Through this enjoyable drama and follow-up activity, the students 
were familiarising themselves with the connections among text, code, and the 
physical and communicative activity they represented. 

Emergent coders with some experience can be asked to look at a story like 
Can I Come In? without access to the code, and then to generate their own 
versions of the story — or a segment of it — with relevant sprites, costumes, and 
code. This would build confidence and fluency in coding at their current level 
of competence. The kinds of extending tasks for advanced learners might be 
examining the code for Can I Come In? to determine how the changes in facial 
expression were managed when the facial features were discrete sprites detached 
from the blank base face of the sprite, and then attempting to incorporate this 
approach in their own stories. Further extension options are becoming available 
through analysis of the strategies of advanced Scratchers (Hagge, 2018, 2021). 
An expanded repertoire of learning experiences along such lines would always 
need to be strategically drawn upon to support the students’ commitment to 
creating a personally meaningful narrative experience. 


Re-configuring research in classroom integration of coding 
and literacy 


The potential of integrating coding into ELA education not only offers a pro- 
ductive response to international imperatives for developing students’ coding and 
computational thinking as a core competency across curriculum areas, but it 
also offers a more enjoyable and agentive role for students in engaging creatively 
with digital animation as an increasingly popular form of multimodal literate 
practice. Studies introducing story coding with Scratch to date have drawn on 
a few examples of stories coded by the school-age participants. The significant 


Advancing animated story composition through coding 191 


disparity between these stories and those created independently by school-age 
peers who are coding enthusiasts, reflects the kinds of development that are 
needed in research and practical pedagogy to realise the potential of integrating 
coding in ELA education so that students’ achievements in coding and narrative 
authoring align with curriculum expectations. Advancing animated story com- 
position through coding in school curricula could benefit from transdisciplinary 
research that involves joint participation by researchers in computing and literacy 
and language arts education, and design-based research (DBR) that includes 
teachers as research partners (Anderson & Shattuck, 2012). 

Transdisciplinary research is very different from ‘inter’- or ‘multidisciplinary’ 
research (Halliday, 2003 [1990]). The latter implies research within disciplines 
while building bridges between them, and/or assembling the research efforts into 
a ‘collection’; transdisciplinary research seeks to transcend disciplinary bound- 
aries to achieve the integrated focus necessary for investigating pedagogies at 
the intersection of coding and narrative composition. This means that research- 
ers need to commit to intensive critical engagement in discourses beyond the 
discipline(s) that they were trained in, have become their principal career focus, 
and through which their prestige is established (Unsworth, 2008). We believe 
commitment to a transdisciplinary approach is a priority in building credible 
curriculum proposals and pedagogic approaches that will persuade literacy teach- 
ers to invest in the transdisciplinary professional learning that will enable them 
to optimise coding as an important dimension of multimodal literacy in ELA. 

Assuming a transdisciplinary approach, bringing the achievement of novices 
in animated story creation and coding competence to a level commensurate 
with relevant curriculum requirements could be effectively advanced by a DBR 
approach that extends well beyond the seven-week interventions of existing 
studies. The purpose of DBR is to build theory and to improve practice within 
authentic education environments (Anderson & Shattuck, 2012; Reimann, 
2011). It involves teachers and researchers partnering in response to a commonly 
agreed issue of significance to produce iterative cycles of intervention, planning, 
and implementation within specific education contexts. Teachers and researchers 
work collaboratively to build understanding of the learning context, and to select 
and/or design interventions based on professional and research literature, theory, 
and practice. During the iterative cycles of DBR, the data that are collected 
and analysed after each intervention are used to refine and improve the next. 
This kind of longitudinally oriented, transdisciplinary informed, DBR approach 
would enable the same cohort of students to participate in successive cycles of 
coding animated narratives. 

The initial cycle in introducing Scratch might deal with developing and/or 
consolidating students’ multimodal authoring in terms of their visual representa- 
tion of characters’ attitudes, constructing varying points of view for audience 
engagement in observing, visually interacting with and aligning with or adopting 
characters’ perspectives. This entails exploration of Scratch sprites, their existing 
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costumes, affordances of their construction for modification, drawing in Scratch 
to modify facial expressions, importing or drawing new sprites, and informal 
exploration of some of the Scratch coding functions to animate selected sprites 
to complete various actions. At the same time, teachers could show examples 
of Scratch stories, highlighting multimodal authoring techniques and informally 
indicating how the Scratch blocks enable them, as well as modelling the creation 
of short within-story sequences. 

The second cycle might involve consolidating students’ knowledge of and 
capacity to deploy basic function blocks in Scratch, and teacher/student collab- 
orative story building with high-level teacher scaffolding, followed by teacher 
modelling of original story creation. After this, students may work in pairs or 
small groups to generate their own stories with teacher guidance and support 
using storyboards to plan in detail. The emphasis in the initial story creation 
phase would be on creating an optimally engaging story within a modest reper- 
toire of coding competence. 

After the first round of stories have been coded, they might be shared and 
examined for potential refinement, both in terms of alternative and more effi- 
cient coding options to achieve the same story features, or editing the story 
to include more complex events, character actions and responses and/or con- 
struction of the audience into the story through visual point of view options 
and/or interactive fiction (see Chapter 10). Follow up story composition may 
involve more independent work by students individually, in authoring teams and 
as part of online communities. Learning progressions along these lines might well 
extend over at least one semester or over a full year. 

The potential of integrating coding and literacy is evidenced by the impressive 
literary quality of coded animated narratives created by school-age children very 
largely independently of their school experience. Popular block coding platforms, 
like Scratch, have been instrumental in allowing early entry by these children into 
the changing literacies of the digital communicative world of the 21st century. 
Dissemination of these opportunities to the broad and diverse populace of school 
systems entails multitudinous complex challenges. Among them, the issues facing 
further educational research and development of curriculum and pedagogy for 
integrating coding and literacy call for the de-siloing of disciplinary endeavours so 
that new epistemologies based on transdisciplinary commitment can emerge. This 
de-siloing is essential to underpin the pedagogic practices needed for the universal 
preparation of children to engage productively with coding as a literate practice 
among the constantly evolving multiple forms of literacy for digital futures. 
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DIGITAL INTERACTIVE LITERATURE 


Introduction 


The integration of literature and new technologies continues to evolve, main- 
taining the longstanding trajectory of literary adaptation and innovation with 
the ever-changing development of film and communication technologies. As 
digital technology has become a defining aspect of contemporary global soci- 
ety, we experience the ongoing recontextualisation of classic and contemporary 
literature afforded by digital technology development, as well as new literature 
specifically created to leverage those technological affordances. Students grow- 
ing up in these digital communication environments have an increasing range of 
different kinds of digital interactive literature (DIL) available to them, including 
the same stories — such as the Harry Potter story world (Rowling, 2017) — not 
only as books and movies, but also as multiple video games, and even now as 
augmented reality games — such as Harry Potter: Wizards Unite (Warner Brothers 
and Niantic, 2020). At the same time, students can experience new interactive 
narratives created with and for virtual reality and augmented reality applications. 

The literacy competencies entailed in engaging with and critically appreciat- 
ing these multiple interactive digital formats will need to continue to evolve. 
For teachers to maintain a socially responsible literacy pedagogy in this con- 
text, frameworks will be required for examining how multiple options for digital 
interactivity are related to the interpretive possibilities of the stories. To develop 
such frameworks, we first need to specify the modes through which interactiv- 
ity can occur, such as visual, verbal and bodily modes, and the available options 
within each. From this cartography of interactivity potential, we can map the 
choices or combinations thereof that create different types of interactivity. Then 
we can examine how different types of interactivity contribute to story meaning, 
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enabling a critical appreciation of the overall co-patterning of different types of 
interactivity and its role in prompting interpretive engagement in the story. 

To pursue this agenda, we adapt the approach of systemic functional linguis- 
tics (SFL) and semiotics, in which complex interacting networks of options for 
meaning making within representational modes (such as language and image) are 
mapped and instances of meaning making are described in terms of the choices 
made from such networks (Halliday, 2013). In this chapter we, therefore, propose 
a framework, drawing on the SFL or ‘system network’ approach (Martin, 1987), 
to map the available options in three core dimensions of interactivity (imagic, 
bodily, and verbal) and their narrative functions as peripheral or integral to the story. 


Networking dimensions of interactivity and narrative function 


The main systems for mapping the dimensions of interactivity to their narrative 
functions are shown in small caps in Figure 10.1. 

The superordinate system of INTERACTIVITY encompasses the main contribut- 
ing systems (DIMENSIONS and NARRATIVE FUNCTIONS) in a left-facing brace. The 
brace indicates that as one proceeds from left to right through the network to 
describe a particular instance of interactivity, it is obligatory to select further 
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FIGURE 10.1 A network base for mapping interactivity in digital literature 
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options from both main systems. Within DIMENSIONS, the initial options are 
imagic, bodily, and verbal, which are located as discrete options on a straight-line 
bracket. Such a bracket means that only one of these options can be selected, 
however, the combination of the left-facing brace and the straight-line bracket 
means that one or more of the options can be selected simultaneously. Hence, 
both imagic and bodily can be selected to account for digital narratives where 
imagic and bodily interactivity co-occur. In describing a particular instance of 
interactivity, one must select from the features of the DIMENSIONS and the NARRA- 
TIVE FUNCTIONS systems. The NARRATIVE FUNCTIONS features of integral and periph- 
eral are dichotomous choices, as indicated by the straight-line bracket. So in one 
instance, the combination of imagic and bodily interactivity might be peripheral 
to the narrative, while in another instance it may be integral. 


Imagic, bodily, and verbal interactivity 
Imagic interactivity 


Further imagic options include format, dimensionality, view, and mobility (Figure 10.2). 
The format may be virtual reality (to be discussed in the section titled “Interactivity 
in virtual reality story apps’) or an illustration. Dimensionality in virtual reality is 
necessarily 3D, and 2D in illustrations. What the audience sees in the representa- 
tion we refer to as view. In 2D illustrations, the view is fixed to that chosen by 
the illustrator, but in 3D virtual reality, the view is navigable by the user. Clearly, 
the options within format, dimensionality, and view do not freely combine. For 
illustration, the dimensionality is necessarily 2D and the view fixed, whereas for 
virtual reality dimensionality is 3D and the view may be navigable. In the final 
imagic option mobility, there are three further sub-options. The first is whether 
image elements are manipulable by the user — and if so, whether this is free 
manipulation or constrained within pre-set parameters, such as a character being 
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movable in a set pattern. In Figure 10.2, the dash option in the square bracket 
with manipulable indicates the ‘not manipulable’ option, and, similarly, the dash 
in the pre-set bracket indicates ‘not pre-set’. The second sub-option is whether 
the image elements are moving or stationary, and the third is whether they are 
animated or not. For example, a character may be stationary, but animation keeps 
the eyes blinking. 

The left-facing brace following ‘imagic’ means an option from each of format, 
dimensionality, view, and mobility must be selected. Similarly, the left-facing brace 
following ‘mobility’ means that choices must be made from each of the three 
subsequent sub-systems. For the first two of these the combined brace and square 
bracket means that either or both options can be selected. For example, within 
any scene, some image elements may be manipulable and others not, and some 
may be moving and others not. 

Characters in 2D and 3D can invoke interaction through, for example, a direct 
frontal gaze at the audience, a frontal view of arms outstretched directly toward 
the audience as a greeting, through the construction of the audience’s point of 
view as if they were a character in the story, or through an ‘over-the-shoulder’ 
positioning of the audience to see along with the character. The represented 
physicality of 3D characters is suggestive of their, at least potential, interactant 
role with the audience, while inanimate 3D objects suggest they can be handled, 
circumnavigated, climbed upon, etc. In addition, the ability of the audience to 
navigate to different views within a 3D environment is a form of interaction with 
a particular setting. Animation such as eye movement, head nodding, hand wav- 
ing and so on, can invoke audience interaction. The movement (displacement) of 
participants invokes interaction through ‘tracking’ by the audience. 


Bodily interactivity 


Audience bodily interactivity involves various kinds of touch on the screen or 
manoeuvre of the device. These are sometimes cued by verbal and/or visual on- 
screen notifications. Verbal interactivity can also be a discrete system, which can 
be oral or in print. These options and their further sub-options are added to our 
system of DIMENSIONS of interactivity in Figure 10.3. 

In DIL where the imagic format is illustrated, there is usually limited interac- 
tivity. This is common in interactive versions of classic children’s literature such 
as Alice for the iPad (Carroll, 2016), which is a re-versioning of Alices Adventures 
in Wonderland by Lewis Carroll (1865). The iPad version maintains the original 
text and illustrations. All images are 2D with the audience view fixed. There are 
two ways the reader can interact with the images. Firstly, characters as stationary 
images, are manipulable by touch (tap or drag) but within pre-set parameters. For 
example, a tap on the character results in the caterpillar moving its hookah to its 
mouth and back in a repetitive pattern, the Duchess’s baby rocking and crying, 
and the Mad Hatter’s head bobbing back and forth. When the image of Alice 
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FIGURE 10.3 Dimensions of interactivity 


automatically shrinks, it is possible to drag to restore Alice to her original size. 
Secondly, objects can be dragged anywhere on the screen, such as the ‘Drink Me’ 
bottle that shrunk Alice, Alice’s comfits (lollies), little cakes, mushrooms, the 
Queen of Hearts’ crown, and the playing cards that float over the screen. 

Similar limited imagic interactivity is possible in digital interactive versions 
of classic children’s picture books such as Jemima Puddle-Duck (Potter, 2013), 
which is a re-versioning of Beatrix Potter (1908) story. In this app, animation 
involves the characters blinking, and while most are stationary and manipulable 
within pre-set parameters, there is an occasion where Jemima is moving (flying) 
and can be dragged all over the screen. These are also the most common forms 
of imagic interactivity in many contemporary children’s interactive story apps. 
Some include other touch techniques such as swipe, pinch (pulling the thumb 
and forefinger together) and reverse pinch (Naji, 2021). 

Interactivity can also occur through manoeuvring the device (shake, tilt or 
rotate). For example, in The Cloud Factory (Stokes, 2016), the user is invited to 
rotate the iPad 90 degrees to pour milk from a bottle into the bowl of ingredients 
for cloud making and later to shake the iPad to mix the ingredients. On-screen 
cues to the bodily interactivity are frequently provided, such as the question 
mark inside a yellow disk at the bottom of the screen in The Cloud Factory 
(Stokes, 2016). If the viewer does not take action a white hand points to the disk 


Digital interactive literature 201 


SHAKE THE 


DEVICE! 


FIGURE 10.4 On-screen notification for bodily interactivity 
Stokes, 2016 


and the words “PUSH HERE FOR HELP” appear. Upon doing so, a verbal 
and visual cue to the bodily action required appears (Figure 10.4). Similar cues 
occur in other apps such as the on-screen “HINT” in ‘The Heart and the Bottle 
(Jeffers, 2010). But not all opportunities for bodily action are cued. In The Cloud 
Factory, for example, the use of swipe to remove the cow from the first screen is 
not cued and in Alice for the iPad and Jemima Puddle-Duck, there are no cues to 


> 


use various touch techniques. 

The Thief of Wishes (Markowska, 2017) is a children’s story with principally 
stationary images that include some limited animations such as characters’ eye 
movements, arm movements, etc. and background animations such as leaves 
floating onto water. Apart from this, the interactivity consists of on-screen 
‘choose your own adventure’ verbal text interpolations offering usually two 
options for advancing the story. Many contemporary digital interactive novels 
for older children are verbal only. The Hero of Kendrickstone (Wang, 2015) is an 
interactive fantasy novel of the ‘choose your own adventure’ type with no images 
or sound. At frequent intervals, readers are presented with three or four options 
from which to choose to advance the story. Examples of other verbal only stories 
include Sixth Grade Detective (Hughes, 2016), an interactive novel for early teen- 
agers and Running Away (Ardeshir, 2019) in the format of an interactive social 
media chat dialogue. 


202 Texts and digital semiotics 


Interactivity in virtual reality story apps 


There are two main types of virtual reality (VR): non-immersive and immer- 
sive (Coban, 2021; Freina & Ott, 2015; Lee et al., 2020). Non-immersive VR 
(NVR) is a desktop-based technology that simulates a 3D virtual environment 
and provides the user with some degree of telepresence (Steuer, 1992, p. 76), 
defined as a mediated perception of ‘being there’ in that virtual environment. 
In NVR, although the images look three dimensional, they are rendered on a 
2D flat or curved screen, so the user retains the perception that s/he is separated 
from the virtual environment. 

Immersive virtual reality (IVR) involves wearing a head-mounted display 
(HMD) over the user’s eyes, which tracks user position and projects stereo 
images for each eye corresponding to where the user is looking in the virtual 
environment, so that users perceive themselves as located within the virtual envi- 
ronment (Jensen & Konradsen, 2018; Pottle, 2019). The HMD obviates any 
perception of the immediate material environment, so the user experiences an 
all-round perception of physical presence in the virtual environment (Slater & 
Sanchez-Vives, 2016). 

NVR is experienced with many modern videogames and virtual worlds 
such as Second Life, as well as VR versions of literature such as Sherlock Moviebook 
(Doyle, 2014), a rendering of the Sherlock Holmes story, The Red-Headed League 
(Doyle, 1892), and for young children, the Bookful rendering of The Tale of Peter 
Rabbit (https://bookful.app/books/the-tale-of-peter-rabbit/) from the original 
by Beatrix Potter (1987). These stories exemplify very different approaches to 
NVR in digital narrative technique. In the Sherlock Moviebook story, Jabez Wilson 
comes to Sherlock Holmes claiming he has been wronged by a mysterious league 
of red-headed men, which was founded by an eccentric red-headed millionaire 
who wished to provide for other red-headed men by offering them easy jobs for 
high pay. Wilson acquired one such job, copying out the encyclopedia for four 
pounds a week. However, after some months, a sign on the office door announced 
that the league had been disbanded. Wilson was encouraged to apply for the job 
by his very efficient new assistant, Vincent Spaulding, who was happy to work for 
half wages. Wilson reveals that Spaulding spent hours in Wilson’s cellar each day, 
developing his photographs. Holmes determines that when ostensibly developing 
photographs, Spaulding was tunnelling under the shop to get access to a nearby 
bank vault and that the job with the red-headed league was a ruse to get Wilson 
away from the shop until the tunnelling was completed. 

The story is portrayed on screen as an illustrated novel with images and text 
on each page. Rotating the iPad displays the images in full screen with audio 
reading of the story. Swiping upwards reveals thumbnails of each page to enable 
re-reading/viewing at any stage in the story. Some images are user-navigable 
NVR environments while others are NVR movies, which are not manipulable 
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by the user, and other illustrations are simply stationary images. Swiping on 
the NVR representations enables horizontal and vertical navigation around the 
three-dimensional virtual environment. For example, the view of Wilson ini- 
tially recounting his experience to Holmes shows them in lounge chairs opposite 
each other but swiping enables the user’s gaze to shift around so that nearly all of 
Holmes’s room can be viewed. 

The animation of the 3D characters in NVR enhances the user’s experience 
of telepresence. For example, when Wilson sneezes after taking snuff, images of 
droplets of saliva appear over the screen. Touch in NVR also enhances telepres- 
ence. For example, when Holmes and his colleagues enter disused passages to 
the bank vault, swiping not only moves the audience around the disused room, 
but also results in wiping dust off one of the boxes stored therein (p. 24). Hence, 
the NVR movie segments and the 2D stationary images contribute significantly 
to user interactivity. 

The Bookful version of the Peter Rabbit story (https://bookful.app/) typifies 
a minimalist deployment of NVR. This story is also portrayed on the screen in 
book format and as the user swipes to turn pages the 2D illustrations convert 
to 3D animations. While the NVR is 3D, the user view is fixed with the single 
view of the virtual environment maintained throughout the story. The animated 
characters repetitively perform the same activities until the page is turned. For 
example, the rabbit’s hop, jump, etc. and Mr McGregor on his knees rhythmi- 
cally and repetitively bends over his garden. Apart from swiping to turn pages, 
there is no bodily interaction with the story. 

These types of NVR we refer to as contained NVR, since they are experi- 
enced entirely within the screen-based environment of the tablet or smartphone 
device. However, another form of NVR is augmented reality (AR). As discussed 
in Chapter 7, AR technologies overlay virtual content, such as animations and 
representations of natural or artificial artefacts or texts, over real-world environ- 
ments, so the user of AR is simultaneously interacting with elements of the 
real world enhanced by computer-generated perceptual information. AR dis- 
plays can be rendered on devices resembling eyeglasses but are more commonly 
accessed with apps using the cameras of smartphones and tablets. 

The apps are cued to activate the AR displays by signals which may be 
embedded in books, and in some cases are projected from the device onto an 
AR receptor such as the ‘Merge Cube’ (https://mergeedu.com/cube). This is an 
inexpensive holographic object which is placed in front of the device and rotated 
to display the AR depictions. The AR activation signals may also be physically 
present in selected real-world environments, or they may be activated by the 
Global Positioning System (GPS) on the device. We can now extend the format 
system to include more detailed options for imagic interactivity (Figure 10.5). 

Adaptations of literary classics such as Robert Louis Stevenson’s Strange Case 
of Dr Jekyll and Mr Hyde, Frankenstein by Mary Shelley, and T.S. Eliot’s ‘The Waste 
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FIGURE 10.5 Non-immersive virtual reality contexts for digital interactive literature 


Land’ (Eliot, 2011; Hugli & Kovacovsky, 2010; Morris, 2012), as well as Penguin 
Books’ and Zappar’s interactive versions of Moby Dick, Great Expectations, and oth- 
ers (Farr, 2012), have afforded some status to AR books, in the context of DIL 
(Karhio, 2021; Weedon et al., 2014). Many classic works of children’s litera- 
ture have been re-versioned with AR apps such as the multiple versions of Alice’ 
Adventures in Wonderland (Carroll, 2017, 2018). The paper versions of the books 
have the illustrations cued to respond to AR apps on a smartphone or tablet, which 
then portray 3D animated depictions of the characters enacting set routines. For 
example, in the version published by Ranok (Carroll, 2018), the characters do not 
speak but we see Alice follow the white rabbit down the rabbit hole, drink from 
the bottle which alters her size and swap places around the Mad Hatter’s tea party, 
etc. Many such classic children’s stories follow this AR format, for example, The 
Little Mermaid (Andersen, 2016) and Little Red Riding Hood (Lambert & Butcher, 
2018). Contemporary books for children, such as Toy Story (Kent, 2019) and 
graphic novels for teenagers and young adults such as Chosen Kin (Sparks, 2020) 
also adopt this AR format. Some AR stories, although they will work on the 
smartphone or tablet alone, are specifically designed for use with the Merge Cube 
e.g. 57° North (Mighty Coconut, 2019) and Little Red Riding Hood (PleIQ, 2018). 

The more nuanced deployment of AR as an integral dimension of digital 
literary technique occurs in the experimental literary creation, Sherwood Rise 
(Weedon et al., 2014). This is a derivation from the Robin Hood legend set in 
the years around 2010 in a housing estate in the suburban wastelands of London. 
The privileged elites are pursuing their exploitation of the poor and a band of 
hackers led by a beautiful young woman, Robin, hack newspapers to reveal 
the corruption. The story is told over four days using the mobile phone AR 
app, emails and four newspapers, which are in three different versions. The AR 
characters interact with the user via email. Which version of the paper the user 
receives depends on how s/he has responded to the efforts of Robin and the 
‘Merry Men’ in championing the causes of the poor, so the reader becomes part 
of an illegal group of outlaws, becomes implicated and takes sides. The develop- 
ment of the positions taken by the user affects how they progress with the story. 

The finessing of AR techniques to build interactivity in literary narrative 
continues to develop with ongoing research such as the innovative AR picture 
book for adults, Saints of Paradox (Tavares, 2019). The main character, Elza, is an 
elderly woman who mourns the death of her lover, Euclides, who disappeared 
during the 1964 Brazilian coup d’etat. For 50 years, she has been immersed in 
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grief. She initially searched for her lover but has since retreated into a deeply 
religious existence, bordering on obsession and madness. She watches an anti- 
quated television that repetitively plays propaganda from the 1960s. She becomes 
fixated on the day that her lover disappeared. One day, she accidentally breaks 
a photograph of Euclides, which triggers her vision of the place where he had 
been imprisoned. She sees him dance, but suddenly he is shot. 

The book consists of ten richly elaborate pictures. When the pages are 
scanned, the user can select narrations for each picture from one of three differ- 
ent Saints: The Mother of Benevolence, the Father of Pragmatism, or the Father 
of Orthodoxy. Each saint has a distinctive monologue, soundscape, video, and 
animated interpretation that plays out over the printed picture, the AR depic- 
tion for each narrator changing the printed picture to form its own distinctive 
remaking of the image and hence divergent interpretation of the scene. Hence, 
the user interacting with and responding to AR depictions is contributing to the 
instantiation of the particular narration that is derivable from the potential nar- 
rative progressions. The vision of her lover from her youth has a dramatic impact 
on Elza and the story ends in three different ways, depending on how each saint 
interprets the sequence of events. 

AR in DIL is not only anchored to artefacts, but can also be anchored in 
locations, which can be specific or generic (wherever the user happens to be). For 
example, the 2017 winner of the New Media Writing Prize, The Cartographer’s 
Confession (Attlee, 2017) is a fictional narrative set in specific locations in London. 
It tells the story of Thomas Andersen, whose Norwegian mother Ellen brought 
him to London in 1945 as a small child in a suitcase. Since Thomas’s father was 
German, Ellen had her head shaved in Norway. London was a bewildering laby- 
rinthine city that neither of them could fathom, resulting in a life of anxiety and 
trauma. To survive and make sense of their environment Thomas began chart- 
ing the city through the location of places they became attached to. From this 
beginning, Thomas later became a cartographer. The AR story unfolds as users 
walk around three city locations. More story material becomes available as the 
users move between the three sites where the story takes place. The storying is 
via voice, video, text, historical photographs of 1940s London, new illustrations, 
sound, and music (Parezanovié, 2019). 

In Silent Streets: Mockingbird (Cobbett, 2018) the AR location is generic. This 
is a mystery adventure in the tradition of Sherlock Holmes set in Newport in 
Victorian times in which a detective has to find the killer of a local boxing cham- 
pion. Augmented reality depictions are shown in the user’s own environment 
and can range from inspecting various ‘clues’ on an AR depicted writing table to 
examining an AR depicted corpse in a morgue. The story follows a ‘choose your 
own adventure’ format, where user choices affect the progress and conclusion of 
the story. There are many other similar AR adventure stories such as Anomaly: 
Clandestine (Zenfri, 2015), which involves the user defending his/her own envi- 
ronment against invading aliens. For young children examples of generically 
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located AR DIL are typified by those available via the Wonderscope app (https:// 
wonderscope.com/). Some are derivations from traditional tales such as Little 
Red — The Inventor (Bora, 2018). In this twist on the traditional Red Riding Hood 
tale, the app user works with Little Red to overcome her initial deception by 
the wolf and to assist in eventually saving her grandma and capturing the wolf. 
The app prompts the user to find a well-lit and flat location in the user’s environ- 
ment on which the AR world is depicted. The story employs voice activation 
to involve the user interacting orally with Little Red and hot spots to enable the 
user to assist Little Red in finding and activating crucial objects, helping her in 
completing difficult puzzles, and providing encouragement when she becomes 
disheartened. The AR affordances are deployed powerfully to achieve the kind 
of user interactivity that promotes empathy with the depicted characters. 

In IVR stories, experienced via HMDs, the concept of the book or even the 
digital visual facsimile of a book is completely discarded as in the IVR version 
of Wolves in the Walls (Fable Studio, 2019) or almost completely discarded as in 
the IVR story of Tara’s Locket (Big Motive, 2017). Tara’s Locket, nevertheless, is 
described as “an illustrated storybook adventure in a stunning virtual world.... 
inspired by the landscapes, stories and folklore of Ireland’s dramatic Atlantic 
coast” (https://www.wearvr.com/apps/taras-locket-a-vr-story-for-children). 

Intended for children (four to eight years), the story involves the user in Tara’s 
adventures of bringing her family safely back together from their perilous expe- 
rience working in their fishing boat. The virtual world is set up so that users can 
explore the 3D environments without bodily navigating through virtual space. 
The creators “designed Tara’s world to be explored simply by looking around 
and designed the interactions to be triggered by the user’s area of attention” 
(https://www.bigmotive.com/blog/taras-story-a-childrens-book-brought-to- 
life-in-vr/). The story is progressed through interactivity cued by visually pre- 
sented ‘hints’, that draw attention to outlined participants which, once selected, 
become full 3D story components. 

The creators of Tara’s Locket were very much aware of the role of literary 
picture books in introducing children to active reading and growing facility in 
interpreting text, so this aspect of the book format was maintained: 


Conscious also of the fact that our users want to interact and not just 
watch, we created the interludes around flowing sentences that reveal 
themselves as the user follows the text and guides them to the conclusion. 
For users too young to read the words this ‘flowing sentence’ model is use- 
ful in reinforcing some very simple literacy fundamentals, such as reading 
from left to right. As a rule of thumb, VR is best when it relies on the least 
amount of text, but it can work to enhance the experience when used 
sparingly to change the pace and progress the narrative. 
(https: //www.bigmotive.com/blog/ 
taras-story-a-childrens-book-brought-to-life-in-vr/) 
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FIGURE 10.6 Flowing sentence interludes in Tara’s Locket 
https://www.bigmotive.com/blog/taras-story-a-childrens-book-brought-to-life-in-vr/ 


This aspect of the IVR story is indicated in Figure 10.6. 

The IVR experience of Wolves in the Walls (Fable Studio, 2019) is based on 
the award-winning picture book of the same name (Gaiman, 2003). In this story 
for young people 13 years and older, young Lucy hears the sounds of wolves in 
the walls of the family home. Her family dismisses these claims suggesting that 
harmless creatures that often inhabit the walls of old houses are responsible. Lucy’s 
claims prove to be correct and her resolution for the dilemma this causes for the 
family constitutes a highly engaging, open-ended literary experience. Users can 
join Lucy in exploring the very detailed 3D representation of the family home 
and its contents. Users who have touch controllers can manipulate these to hold 
a camera passed to the user by Lucy to collect evidence of the wolves’ presence. 
This means the user can also retract photos from the polaroid camera and pass 
them to Lucy, take a magnifying glass from Lucy to inspect the photos, and mov- 
ing around the attic and looking under furniture, the user can pick up fallen pho- 
tos. Users can also accept a crayon from Lucy to write their names in the story 
credits. The user is both a participant and an observer as the story progresses. 

Through dialogic and visual interaction with Lucy, users experience moving 
through the rooms of the house, hearing the household sounds (including those 
of the wolves), bending under tables, etc., and experience their presence in the 
story world as they support and assist Lucy. The IVR experience of interactivity 
in Wolves in the Walls involves the manoeuvrability of the user’s body as well as 
the ability to handle and manipulate objects, while the visual interactivity with 
3D depictions affords navigable views, open-ended manipulation of objects, 
and interaction with animated moving participants. This kind of multi-sensory 
immersive active participation in the story world provides a very distinctive basis 
for interpretive responses to such narratives. 
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FIGURE 10.7 Dimensions of interactivity with non-immersive virtual reality contexts 
expanded 


We can now complete our mapping of options within the DIMENSIONS system 
for interactivity as shown in Figure 10.7. 


Narrative functions of interactivity 


We now consider the contribution of the patterns of choices from the DIMEN- 
SIONS of interactivity to their NARRATIVE FUNCTIONS. The initial consideration is 
whether a particular instance of interactivity is peripheral or integral to the story 
(Figure 10.8). 

In Alice for the iPad (Carroll, 2016) many instances of interactivity — such as 
tapping resulting in Mad Hatter’s head bobbing or the Duchess’s baby rocking and 
crying — are peripheral, as is the tapping in Jemima Puddle-Duck (Potter, 2013), 
which results in quacking or the noises of other animals. This is a prominent 
form of interactivity in digital interactive versions of picture books and illustrated 
stories for children. A recent study found that “Most of the AR interactions that 
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FIGURE 10.8 Interactivity as peripheral or integral to the story 
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were found in this study consisted of taps and pokes of AR elements on the page 
resulting in squeals and other noises” (Green et al., 2019, p. 372). However, this 
study did find some interactivity integral to the stories. For example, in TJ and 
the Beanstalk (Pai, 2017) the user assists in cutting down the beanstalk by swip- 
ing in AR and in The Adventures Suit (Zappar, 2015) the picture book shows 
the little boy in reality in his dress-up suits that he imagines transport him on a 
variety of adventures, while the AR shows the boy on the imagined trips (Green 
et al., 2019, p. 371). The Heart and the Bottle (Jeffers, 2010) has similar examples 
of interactivity that are integral to the story (Zhao & Unsworth, 2017) and we 
have noted further examples in previous sections such as rotating the device to 
pour the milk in The Cloud Factory (Stokes, 2016). 

We now examine three ways in which the various kinds of interactivity can 
be integral to the story: 


e procedural — activating the functionality of the app; 

e — elaborative — reinforcing the development of plot or characterisation; 

e transformative — influencing the nature of the progress and outcome of the 
story; 


as shown in Figure 10.9. 


Firstly, in our system of NARRATIVE FUNCTIONS (Figure 10.9) we need to 
acknowledge the integral role of procedural interactivity. This has been referred 
to as ‘extra-text’ interactivity (Zhao & Unsworth, 2017, p. 92) because it acti- 
vates aspects of the functionality of the app rather than elements of the story. 
Procedural interactivity includes swiping to turn ‘pages’, hot spots that activate 
thumbnails of all pages enabling the user to jump to different parts of the story, 
and sometimes other features such as a microphone icon as a hot spot that acti- 
vates a ‘read aloud’ function. 

Interactivity that is integral to the story is also commonly elaborative. Such 
interactivity may reinforce or extend the plot or characterisation (Figure 10.10). 

The reinforce function for the plot occurs when the interactivity rehearses what 
has already been conveyed by the images and/or the language. For example, 
in The Cloud Factory (Stokes, 2016) the grandfather says, “Now dear, we need 
to shake the mixture” and the cued interactivity of shaking the iPad reinforces 
this part of the process. The reinforce function for characterisation occurs in The 
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FIGURE 10.9 Integral interactivity 
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Marvellous Machines (Grimm & Fortuna, 2016) when the main character is giving 
a tour of his fantastical robot collection: 


While the book pages describe each of the robot’s behaviours, the AR 
brings the robots to life. For example, one of the robots, The Twinkle- 
Toe-Sprinkle, is described on the pages as a robot that will ‘splash you all 
over’ (p. 13). In the AR, the full extent of the robot’s spraying and spatter- 
ing abilities are demonstrated. 

(Green et al., 2019, pp. 370-371) 


The extend function of interactivity with respect to the plot is exemplified in 
two key scenes in Sherlock Moviebook (Doyle, 2014). The first is when Holmes 
is outside Wilson’s shop tapping on the pavement with his cane (Figure 10.11). 
The text says nothing about where on the pavement Holmes was tapping (“hav- 
ing thumped vigorously upon the pavement with his stick two or three times, 
he went up to the door and knocked” (p.12)). But swiping and tapping on 
the screen activates Holmes’s cane tapping both behind and in front of him 
thus extending the text and providing a clue to the later revealed purpose of 
the tapping as testing the possibility of Spaulding’s secret tunnelling under the 
pavement. The second example of the extend function occurs in the subsequent 
scene, where swiping up on the screen reveals the clue of the dirty knees of 
Spaulding’s trousers not mentioned in the text and not visible in the image until 
swiped upwards. The extend function for characterisation is exemplified in the 
scene following Wilson’s departure from Holmes’s apartment. The text mentions 
Holmes smoking his pipe while considering the case but swiping the 3D image 
of the apartment reveals other aspects of Holmes’s character, which are not vis- 
ible in the initial image depiction, such as the violin on his lounge chair and the 
steel syringe on his desk. 

Similar examples of the extend function can be found in other DIL. The Heart 
and the Bottle (Jeffers, 2010), for example, follows a little girl’s journey through 
grief at the death of her grandfather. She was very close to her grandfather who 
often read her books about the curiosities of life as they both sat in his favourite 
chair. The pages depicting their loving relationship are filled with vibrant warm 
colours. One of the many instances of interactivity that extend characterisation 
occurs when the little girl enters the room with her grandfather’s empty chair. 
Swiping over this page colours the scene with a deep dark blue overlay. 


NARRATIVE ass rocada Sly 
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FIGURE 10.10 Elaborative interactivity 
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FIGURE 10.11 Interactivity extending plot 
Doyle, 2014, p. 12 


From our network of options for interactivity in DIL (Figure 10.12), we can 
now characterise each instance of interactivity by coding the choices it represents 
from the right-most options for each system in the network. 

This helps to indicate to what extent and how the design of interactivity from 
the options within each system contributes to the interpretive possibilities of the 
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FIGURE 10.12 Mapping interactivity in digital interactive literature 


story. To illustrate this, we return to the Sherlock Moviebook interactivity instance 
of Holmes tapping on the pavement outside Wilson’s shop (Figure 10.11) and 
indicate each of the options taken up in the representation of this incident in 
Figure 10.13. In this NVR story, the 3D images are central to the forms of inter- 
activity, but in contrast to most other 3D depictions in the story, in this incident, 
the user’s view is fixed and no amount of swiping or tilting will enable any other 
view. This constraint is strategic in focussing the interactivity on engaging the 
user in the elaboration of the plot. Swiping above Holmes’s hands has no effect 
but swiping behind and in front of him below his hands activates his cane tap- 
ping on the pavement. If the user does not activate the tapping, the visual cue 
appears as shown in Figure 10.11. Only through the interactivity is this clue to 
the mystery accessible to the user. This shows the potential of interactivity design 
as integral to the interpretive possibilities of the story. 

A key issue in the ongoing innovation in DIL is integral interactivity that is trans- 
formative of the story — the extent to which user input determines the nature of 
the story (Weedon et al., 2014). In the AR story Sherwood Rise (http://davemiller. 
org/2019/12/12/sherwood-rise/), we noted earlier that the manner and extent to 
which the user’s actions/choices in the AR sections respond to the efforts of the 
character Robin and the ‘Merry Men’ to ameliorate the conditions of the poor 
influences the way the story progresses. The user’s actions/choices are recorded in 
a database that calculates a score and this determines which of three versions of the 
story newspaper the user receives each day (Weedon et al., 2014, p. 118). 
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FIGURE 10.13 Dimensions of interactivity and narrative functions 


In such stories the scope of possibilities for user influence on the story out- 
come is finite, but a seminal and probably the most open-ended example is the 
user participatory AR drama Façade (Mateas & Stern, 2003). 

In Façade, the user assumes the identity of a friend invited to the apartment of 
a young couple, Grace and Trip, who are experiencing marital difficulties. The 
user necessarily becomes dialogically embroiled in the antagonistic exchanges. 
There are now several versions of this drama: a desktop version with keyboard 
and mouse input, and an alternative desktop version with voice input, as well 
as an IVR version (Dow et al., 2007). Any conversational initiatives of Grace 
or Trip towards the user can be responded to in any way the user wishes, and 
to whatever the user says, Grace or Trip will make a coherent response, so that 
extended conversations can occur. The nature of the conversation determines 
the outcome of the overall exchange and influences the outcome concerning 
the relationship between Grace and Trip. The user, Grace, and Trip can con- 
tinuously move anywhere, use objects, speak, and gesture at any time. Users can 
experience the drama several times to explore how their contributions to the 
conversational exchanges might be varied to achieve different outcomes. The 
DIL available to date does not include user transformative contributions that are 
as open-ended as this. However, Façade indicates the trajectory of this kind of 
development, and along with the innovative transformative options in Saints of 
Paradox (Tavares, 2019) anticipates new forms of transformative interactivity in 
future DIL. 
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The transformative aspect of most currently available DIL occurs as ‘choose 
your own adventure’ interpolations of on-screen text offering two or more 
choices to advance the story. Whether the imagic context is illustrated as in 
The Thief of Wishes (Markowska, 2017) or augmented reality as in Silent Streets: 
Mockingbird (Cobbett, 2018), the story pathway choices are shown verbally and 
selected by touch. However, this form of transformative interactivity occurs 
mostly in text only (or text with background images) in contemporary sto- 
ries such as The Hero of Kendrickstone (Wang, 2015), The Sixth Grade Detective 
(Hughes, 2016), and Running Away (Ardeshir, 2019) and in adaptations of classic 
literature such as Mary Shelley’s (2003/1831) Frankenstein re-versioned as DIL 
by Dave Morris (2012). The transformative multiple pathway/hypertext options 
in the Morris (2012) re-versioning have been hailed as demonstrating innovative 
and sophisticated new narrative techniques consistent with the complex themes 
of the original paper media version (Mills, 2018). 


Cultural challenges: interfacing digital interactivity and 
literary engagement 


Our analysis has highlighted the multiple forms of interactivity in the many dif- 
ferent formats of digital narratives that are now readily accessible online. While 
DIL is proliferating, what is at issue is the kind of engagement with literature that 
is afforded. There remains a chasm separating the integral role of interactivity in 
the innovative artifice of story-world creation and cueing of interpretive audi- 
ence responses in the relatively small number of experimental literary narratives 
such as Sherwood Rise, Saints of Paradise, The Cartographer’s Confession, and Facade, 
and the very large number of adaptations of classic literature as well as contempo- 
rary digital interactive stories for children and youth where interactivity is simply 
procedural or peripheral to the thematic concerns of the story, such as in Toy Story 
(Kent, 2019) and Alice for the iPad (Carroll, 2016). 

In many digital stories and adaptations of literature for child and youth audi- 
ences, where the interactivity is actually integral to the narrative experience, it is a 
combination of the procedural and elaborative interactivity that is largely confined to 
the reinforce function, which rehearses what has already been conveyed by the images 
and/or the language (Figure 10.10), as we discussed in relation to The Cloud Factory 
(Stokes, 2016) and The Marvellous Machines (Grimm & Fortuna, 2016). Some sto- 
ries, such as Silent Streets: Mockingbird (Cobbett, 2018) and classic story adaptations 
such as Little Red — The Inventor (Bora, 2018), are more oriented to the extend 
function — where interactivity augments elements of the plot or aspects of charac- 
terisation, both of which we have illustrated in our analysis of the Sherlock Moviebook 
(Doyle, 2014) segment. Core among the challenges for further enhancing DIL as 
culturally valued activity then, is synthesising multimodal literary authorship and 
the affordances of new digital technologies so that digital interactivity increasingly 
serves key roles in creating the interpretive possibilities of literary narratives. 
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Implications for curriculum and pedagogy 


DIL is essential in school curricula that are oriented to the global trajectory of 
digital futures in multimedia communication and creative expression of con- 
temporary and traditional stories. The iterative re-telling of some of the most 
enduring, entertaining and culturally and personally significant works of clas- 
sic literature for adults and for children in concert with ever-changing new 
technologies would be reason enough for the embracing of DIL in futures- 
oriented curricula. Such re-tellings, leveraging the affordances of new technol- 
ogies, offer innovative, engaging and often challenging re-interpretations of the 
original and subsequent versions of these narratives, as well as emerging innova- 
tive digital narrative forms, all contributing to the ongoing vibrancy of liter- 
ary experience. The societal significance of this curricular attention is further 
supported by the increasing uptake of digital interactive fiction as evidenced 
by the projected global growth in the interactive children’s book market of 
USD755.13 million during 2020—2024 (https:// www. businesswire.com/news/ 
home/20201013006054/en/). 

Curriculum approaches need to foster a discerning appreciation of interactiv- 
ity, which can function in a variety of ways from peripheral amusement to inte- 
gral roles in reinforcing and extending aspects of the narrative. The framework 
we have proposed could guide curriculum recommendations for DIL, and also 
inform the design of classroom learning experiences that focus on the meaning- 
making role of interactivity. Students could be asked to examine a selection of 
instances of interactivity and determine what difference it would make if any 
of these interactivity experiences were deleted. This might also take the form 
of asking to what extent and how interactivity revealed the kind of person a 
particular character was — and how that interpretation of the character would 
be different without the interactivity. Students could be asked to indicate where 
other kinds of interactivity might have been included to illuminate aspects of 
the story, such as a zoom in to show the reaction on a character’s face to certain 
events, or a Zoom out to reveal a detail of the setting that influenced a particu- 
lar action of the character. Further work might probe how the combination of 
dimensions of interactivity functioned to influence the story (as in our discussion 
of the Sherlock Moviebook excerpt) and lead to student suggestions for efficacious 
alternative combinations. 


Recommendations for research 


Transdisciplinary research across fields of digital technology, narratology, semi- 
otics, and literature continues to explore further possibilities for audience par- 
ticipation in DIL, especially in relation to interactivity that is transformative 
of the story development and outcome. While there is also a burgeoning field 
of research exploring a critical poetics of electronic literature (Heckman & 
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O’Sullivan, 2018; Mills, 2018; O’Sullivan, 2019), further research is needed on a 
pedagogically accessible critical poetics of DIL as a basis for developing learning 
experiences that promote engaged, discerning, reflective, and creative responses 
among students. As an initial move, research is needed to build a corpus of stud- 
ies explicating the nature of interactivity and its narrative functions in a range of 
digital literature for audiences from early childhood to youth, to inform educa- 
tors’ selection of stories and their design of learning experiences. 

A transdisciplinary approach to this research incorporating researchers in lit- 
erature for children and youth, multimodal semiotics, digital technology, and 
literature and literacy education can provide complementary perspectives to the 
exploratory framework outlined here mapping the core dimensions of interac- 
tivity and their intersection with narrative function. Immediate implementation 
could provide more detailed and comprehensive explication of literary oriented 
interactivity exemplified in some of the digital literature we have discussed. 
While we have briefly addressed IVR stories, much more investigation is needed 
into the nature and role of the bodily and sensory literacies entailed in differ- 
ent forms of interactivity in these stories and their role in the story experienc- 
ers’ interpretive responses. This kind of explication of the distinctive narrative 
techniques of DIL drawing on the affordances of new technologies will facilitate 
delineation of emerging digital literacy and literary competencies that will sup- 
port socially responsible digital literacy pedagogy into the future. 


References 


Andersen, H.C. (2016). The Little Mermaid — A magical augmented reality book. Books and 
Magic. 

Ardeshir, A. (2019). Running away. Sleeper Cell. [Mobile app] 

Attlee, J. (2017). The cartographer’s confession. Ambient Literature. [Mobile app] https:// 
research.ambientlit.com/cartographersconfession 

Big Motive. (2017). Tara’s locket. Big Motive. [Mobile app] http://www.taraslocket.com/ 

Bora, T. (2018). Little Red the inventor. Wonderscope by Within. [Mobile app] https:// 
vimeo.com/302244305 

Carroll, L. (1865). Alice’s adventures in wonderland. Macmillan. 

Carroll, L. (2016). Alice for the iPad. Oceanhousemedia. 

Carroll, L. (2017). Alice in wonderland. Design Media Publishing. 

Carroll, L. (2018). Alice’s adventures in wonderland. Augmented reality story book. Ranok. 

Coban, M. (2021). Effects of virtual reality learning platforms on usability and presence: 
Immersive vs. non-immersive platform. In G. Panconesi, & M. Guida (Eds.), Handbook 
of research on teaching with virtual environments and AI (pp. 236—265). IGI Global. 

Cobbett, R. (2018). Silent streets: Mockingbird. Funbakers. [Mobile app] 

Dow, S., Mehta, M., MacIntyre, B., & Mateas, M. (2007, November). AR facade: An aug- 
mented reality interactve drama. In Proceedings of the 2007 ACM Symposium on Virtual Reality 
Software and Technology (pp. 215-216). https://doi.org/10.1145/1315184.1315227 

Doyle, A.C. (1892). The adventures of Sherlock Holmes. George Newnes. 

Doyle, A.C. (2014). Sherlock moviebook. Haab LLC. [Mobile app] 


Digital interactive literature 217 


Eliot, T.S. (2011) The waste land for iPad. Faber and Touch Press. 

Fable Studio. (2019). Wolves in the walls. Fable Studio. https://fable-studio.com/ 
wolves-in-the-walls 

Farr, C.R. (2012, May 19). 2-D books are over: Augmented reality breathes new life into 
the classics. Venture beat. https://venturebeat.com/2012/05/19/2-d-books-are-over- 
augmented-reality-breathes-new-life-into-the-classics/ 

Freina, L., & Ott, M. (2015, April). A literature review on immersive virtual reality 
in education: State of the art and perspectives. The International Scientific Conference 
Elearning and Software for Education, 1(133) 10-1007. Bucharest. https://doi. 
org/10.12753/2066-026X-15-020 

Gaiman, N. (2003). The wolves in the walls. Bloomsbury. 

Green, M., McNair, L., Pierce, C., & Harvey, C. (2019). An investigation of augmented 
reality picture books: Meaningful experiences or missed opportunities? Journal of 
Educational Multimedia and Hypermedia, 28(4), 357-380. https://www.learntechlib. 
org/primary/p/182345/ 

Grimm, B.C., & Fortuna, I. (2016). Marvellous machines for silly things. Hidden Worlds. 

Halliday, M.A.K. (2013). Meaning a choice. In G. O’Grady, L. Fontaine, & T. Bartlett 
(Eds.), Systemic functional linguistics: Exploring choice (pp. 15-36). Cambridge University 
Press. https://doi.org/10.1017/CBO9781139583077.003 

Heckman, D., & O’Sullivan, J. (2018). Electronic literature: Contexts and poetics. In 
K.M. Price, & R. Siemens (Eds.), Literary studies in a digital age (pp. 1-28). Modern 
Language Association of America. https://doi.org/10.17613/M6K649S3G 

Hughes, L. (2016). Sixth grade detective. Choice of Games LLC. 

Hugli, M., & Kovacovsky, M. (2010). Jekyll and Hyde augmented reality book. https://www. 
youtube.com/watch?v=wXLwYchMIPA 

Jeffers, O. (2010). The heart and the bottle ISO App). Harper Collins. 

Jensen, L., & Konradsen, F (2018). A review of the use of virtual reality head-mounted 
displays in education and training. Education and Information Technologies, 23(4), 1515— 
1529. https://doi.org/10.1007/s10639-017-9676-0 

Karhio, A. (2021). Augmented Reality. In D. Grigar, & J. O’Sullivan (Eds.), Electronic litera- 
ture as digital humanities: Contexts, forms, & practices (pp. 123-131). Bloomsbury Academic. 

Kent, J. (2019). Toy story — Woody’s augmented reality adventures. Carlton Books. 

Lambert, N., & Butcher, R. (2018). Little red riding hood. Little Hippo Books. 

Lee, V.W., Hodgson, P., Chan, C.S., Fong, A., & Cheung, S.W. (2020). Optimising the learn- 
ing process with immersive virtual reality and non-immersive virtual reality in an educa- 
tional environment. International Journal of Mobile Learning and Organisation, 14(1), 21-35. 

Markowska, L. (2017). The thief of wishes. All Blue Studio. [Mobile app] http://www. 
allbluestudio.com/ 

Martin, J.R. (1987). The meaning of features in systemic linguistics. In R. Fawcett, & 
M. Halliday (Eds.), New developments in systemic linguistics: Theory and description Vol. 1. 
(pp. 14-40). Pinter. 

Mateas, M., & Stern, A. (2003, March). Facade: An experiment in building a fully- 
realized interactive drama. Game Developers Conference (2, pp. 4-8). https://www. 
cc.gatech.edu/fac/Charles.Isbell/classes/reading/papers/MateasSternGDC03.pdf 

Mighty Coconut. (2019). 57° North. [Mobile app] https://www.mightycoconut. 
com/57north/ 

Mills, K.A. (2018). Frankenstein in hyperspace: The gothic return of digital technologies 
to the origins of virtual space in Mary Shelley’s Frankenstein. In C.M. Davison, & 


218 Texts and digital semiotics 


M. Mulvey-Roberts (Eds.), Global Frankenstein (pp. 265-281). Springer. https://doi. 
org/10.1007/978-3-319-78142-6_15 

Morris, D. (2012). Frankenstein. Profile Books and Inkle. 

Naji, J. (2021). Mobile electronic literature. In D. Grigar, & J. O’Sullivan (Eds.), Electronic 
literature as digital humanities: Contexts, forms, & practices (pp. 2010-2016). Bloomsbury 
Academic. 

O'Sullivan, J. (2019). Towards a digital poetics: Electronic literature & literary games. Springer. 

Pai. (2017). TJ & the beanstalk AR story book. Pai Technology. 

Parezanovi¢, T. (2019). Digital fiction and reading cartographers of urbanity. Professional 
Communication and Translation Studies, 12, 72—80. 

PleIQ. (2018). Little red riding hood. PleIQ. [Mobile app] 

Potter, B. (1908). The tale of Jemima Puddle-Duck. Frederick Warne and Co. 

Potter, B. (1987). The tale of Peter Rabbit. Penguin. https://bookful.app/books/ 
the-tale-of-peter-rabbit/ 

Potter, B. (2013). Jemima Puddle-Duck LITE. Squeaky Oak. [Mobile app] https://www. 
appspy.com/app/38398777/jemima-puddle-duck-lite 

Pottle, J. (2019). Virtual reality and the transformation of medical education. Future 
Healthcare Journal, 6(3), 181. https://doi.org/10.7861/fhj.2019-0036 

Rowling, J. (2017). Harry Potter: A history of magic. Bloomsbury. 

Shelley, M. (2003/1831). Frankenstein. Penguin. 

Slater, M., & Sanchez-Vives, M.V. (2016). Enhancing our lives with immersive virtual 
reality. Frontiers in Robotics and AI, 3(74). https://doi.org/10.3389/frobt.2016.00074 

Sparks, M. A. (2020). Chosen kin. iSparked Studios. https://www.isparkedstudios.com/#/ 
chosen-kin/ 

Steuer, J. (1992). Defining virtual reality: Dimensions determining telepresence. Journal of 
Communication, 42(4), 73-93. https://doi.org/10.1111/j.1460-2466.1992.tb00812.x 

Stokes, J. (2016). The cloud factory. Dalmation Studios. [Mobile app] https://appadvice. 
com/game/app/the-cloud-factory-an-interactive-picture-book/ 1103042311 

Tavares, T. (2019). Paradoxical realities: A creative consideration of realismo maravilhoso in an 
interactive digital narrative [Doctoral dissertation], Auckland University of Technology. 

Wang, P. (2015). The hero of Kendrickstone. Choice of Games LLC. [Mobile app] 

Warner Brothers & Niantic. (2020). Harry Potter: Wizards unite. https://www.harry- 
potterwizardsunite.com/ 

Weedon, A., Miller, D., Franco, C.P., Moorhead, D., & Pearce, S. (2014). Crossing 
media boundaries: Adaptations and new media forms of the book. Convergence, 
20(1), 108-124. https://doi.org/10.1177%2F1354856513515968 

Zappar. (2015). The adventure suit. Zappar. https://www.zappar.com/campaigns/the- 
adventure-suit/ 

Zenfri. (2015). Clandestine: Anomaly. Zenfri. http://zenfri.com/clandestineanomaly/ 

Zhao, S., & Unsworth, L. (2017). Touch design and narrative interpretation: A social 
semiotic approach to picture book apps. In N. Kucirkova, & G. Falloon (Eds.), Apps, 
technology and younger learners: International evidence for teaching (pp. 89-102). Routledge. 


17 


CONCLUSION: MULTIMATERIAL 
LITERACIES FOR DIGITAL FUTURES 


This concluding chapter provides a forward-focused perspective of literacy, tech- 
nology, media, and visual communication curriculum in the context of antici- 
pated changes in an AI and automation era, and the increasing sophistication 
of smart technologies. It discusses potential ways to prepare instructional lead- 
ers, teachers, and students for the uncertainties of the future in the context of 
multimaterial literacies — literacy practices that are not only characterised by 
multiple modes, but also by radically different possibilities for the material pro- 
duction of texts. It points to a framework for reconceptualising and evaluating 
technologies and identifying new knowledge and skills that students need for 
success to navigate digital worlds. New skills will be needed to thrive in flex- 
ible, next-generation social practices in an area of artificial intelligence-driven 
media and learning that engages the body and mind differently than compared 
to the past. Pedagogical recommendations are provided for incorporating new 
forms of embodied cognitive engagement with multimaterial texts and neo- 
teric ways of reading and thinking to engage students in materially diverse ways. 
The heterogeneity of multimaterial textual forms in the context of digital media 
development presents new challenges, opportunities, and directions for literacy 
curriculum and research transformation. 

Among many changes, it is certain that the future of literacy practices will 
continue to shape and be shaped by technological shifts, such as artificial intel- 
ligence (AI), that have transformed society, and which are impacting virtually 
all industries this century (Holmes et al., 2019). For example, a decade ago we 
did not have the same opportunities as we have today to use virtual, augmented, 
and extended reality technologies to teach about narratives, or to represent ideas 
virtually in three-dimensional (3D) simulations. Some of these developments, 
particularly augmented reality (AR) technologies, have been buoyed by the 
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widespread accessibility of mobile phones and the social web (see Chapter 7). 
Extended reality technologies, also called XR, are those that extend from virtual 
reality and augmented reality — innovations that have yet to materialise in what is 
expected to be a rapidly developing area this decade, given that these platforms 
have not reached their peak in development (Mills, 2022). 

A useful way to think about new digital practices for literacy is the extent to 
which new practices are simply a modest substitute for a conventional literacy 
practice, such as reading a digital versus analogue clock display, or whether liter- 
acy practices are genuinely and radically transformed. The SAMR model, origi- 
nally designed for assessing computer-based technological tools by Puentedura 
(2003), could be considered as a framework to evaluate new literacy practices in 
digital contexts of use. For example, some technologies, like Dictionary.com, 
function as an almost direct substitute for the print version of dictionary use, 
since the practice of consulting this app from one’s phone is similar to carrying a 
conventional pocket dictionary (see Figure 11.1). 

Then there are technologies that augment print literacy practice, but with 
functional improvements, such as e-books that are lightweight, interactive, search- 
able, shareable, and have compact storage. Next, there is modification — tech 
that allows for significant literacy task redesign — such as mixed reality (MR) 3D 
models, representing concepts in a virtual reality environment, or designing and 
programming a dancing robot. Finally, we can look to redefinition — where digital 
technology allows for the creation of new literacy practices that were previously 
inconceivable. These categories — substitution, augmentation, modification, and 
redefinition — are not inflexible categories, but can be conceived as a continuum of 
practices that have greater or lesser resemblance to conventional forms of literacy. 

Educators need to understand how changes to information-based societ- 
ies, and the expansion of technological developments for communication, have 
given rise to new media skill sets that will be essential for the productivity of 
nations. It is predicted that in the next two decades, half of the current occupa- 
tions in countries of the OECD will experience dramatic digital disruption, 
while new occupations that are currently unimagined will emerge (PwC, 2015). 


Á Redefinition 
Á Modification 
Á Augmentation 
Substitution 


FIGURE 11.1 SAMR model 
Adapted from Puentedura (2003) 
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These changes will involve new digital media practices that require distinct forms 
of embodied cognition with new material and technological networks. These 
will involve multimodal and multimaterial texts in social contexts that differ in 
fundamental ways to the past. While literacy, media, and technology educators 
have demonstrated great agility in response to digital change in the past, the rate 
of change is speeding up, and competition in global economies is heightened for 
schools, post-compulsory education, and workforce readiness. 

The undeniable influence of digital and global communication environments 
has been recognised since the end of the last century (New London Group, 
2000), which is important in rethinking literacy practices in schools and soci- 
ety. Most online spaces require students’ proficiency with multimodal literacy 
practices, where students use new platforms to combine words, images, sound, 
body movement, and spatial configurations in new ways to shift meanings across 
modes and media (Mills & Brown, 2021). Creativity remains a key facet of future 
skill sets, which can be exemplified, for example, in computer-aided design or 
CAD (Harris & de Bruin, 2018). 3D printing involves multimodal making of 
tangible signs that count as a form of hybrid literacy. For example, a 3D-printed 
festive tree ornament is a material sign that communicates or stands for cultural 
meanings that can potentially be understood by others. Creative multimodal 
designing associated with additive manufacturing and rapid prototyping tech- 
nologies is now taught in education contexts, including 3D printing, screen 
printing, and laser cutting (Ford & Minshall, 2019). 


Materiality of representation: emerging directions 


Significantly, the materiality of textual practices has undergone dramatic shifts in 
recent decades, not just the visual displays. As we have argued throughout this 
volume, the materiality of the medium matters to literacy practices, whether of 
writing with a pen on paper, typing on a computer keyboard, swiping a touch- 
screen, or painting in the air using VR game controllers. Recently, social semi- 
otic theorists have acknowledged in their third edition of Reading Images: The 
Grammar of Visual Design (Kress & van Leeuwen, 2021), that their visual image 
analysis across many of the chapters of their volume has tended to “abstract away” 
the diverse material aspects of the images, whether of paintings, photographs, 
web pages, drawings, collages, menus, billboards, and so on (p. 224). In so doing, 
they suggest that they followed a Renaissance tradition associated with artistic 
practices of da Vinci and Michelangelo, in which the material aspects of a paint- 
ing were considered a secondary concern, left to the assistant, while the artists’ 
conception of a depicted artwork was most essential. However, in their subse- 
quent chapters and editions, they have given special attention to the material 
dimensions of visual production, such as the embodied performances of singers, 
dancers, and actors, which they see as equally important to the design of the 
music score or a play script in meaning making. 
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Today, the materiality of texts is changing as schools introduce teaching and 
learning using additive manufacturing and 3D printing applications that involve 
modelling data associated with fabrication industries. Rapid prototyping tech- 
nologies, 3D hubs, and fablabs have been used in schools to inspire creativ- 
ity (Ford & Minshall, 2019; Kostakis et al., 2015), often leading to learning 
outcomes and diverse material forms of three-dimensional, multimodal designs 
that are to be interpreted as signs imbued with meaning. There is a need for 
new multimodal literacy curriculum and teaching resources that simplify the 
incorporation of low-cost additive manufacturing and 3D printing to support 
students’ knowledge in these advancing areas of technology change, to induct 
students into the tangible media that will figure in life and work in the new 
digital age (Trust & Maloy, 2017). 

While material shifts from reading and writing mostly on paper to mostly 
on screen began decades ago, digital communication environments continue to 
change in hybrid ways. For example, the materiality of texts popularised in the 
late 1990s and early 2000s were often displayed using a desktop computer — 
instant messaging, chat rooms, email, internet searches, web pages, and PC-based 
video games. By the 2010s, the widespread accessibility of mobile phones and 
tablet technologies, including handheld touchscreens, provided further impe- 
tus for the changing materiality of textual practices that could occur anywhere, 
anytime, and which involved different forms of haptic engagement and social 
contexts (e.g. texts displayed on a phone used on a public bus). At the same time, 
Web 2.0 became important as everyday users could easily share moving image 
texts via social media and video sharing sites, while multiplayer games provided 
real-time interaction in connected game spaces. 

More recently, the materiality of virtual reality headsets has made three- 
dimensional narrative representation accessible, such as watching 360-degree 
films, virtually spray-painting murals in simulated urban spaces, and playing vir- 
tual reality games (see Figure 11.2). 

Literacy educators should be aware that in the next ten years, virtual, aug- 
mented, and mixed reality technologies are predicted to reach a peak growth 
stage of affordability and accessibility, offering new potential for learning and 3D 
meaning making in narrative and non-narrative textual formats. Virtual reality 
technologies provide a digitally simulated environment, with the user wearing 
a head-mounted display for full immersion. As discussed in Chapter 7, MR 
merges virtual and real in interactive ways (e.g. Microsoft Hololens; Mills, 2022), 
while AR overlays virtual content in real locations, typically viewed through an 
internet browser, and often supported by smartphones (see Figure 11.3). 

Literacy teachers in the future will gain proficiency with these technologies 
that are able to give simulated experiences of places in the real world, past or 
present, that can enliven historical fiction, and which are not possible to explore 
in real life (see Chapter 4). These technologies are particularly useful for explor- 
ing or creating three-dimensional representations, and for supporting abstract 
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FIGURE 11.2 VR community subway mural painted by 11-year-olds in the game 
Vive Spray 2 


FIGURE 11.3 AR dinosaur photographed by Year 6 student with Google browser 
and tablet 
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FIGURE 11.4 Mixed media e-sculpture using Arduino kits at Toledo Museum of Art 


thinking and communicating concepts in experiential ways (Fernandez, 2017). 
An exponential rise in the use of these extended reality technologies is antici- 
pated to influence all levels of education (Mills & Brown, 2021). 

Material representational forms that were traditionally associated with sci- 
ence, technology, engineering, and mathematics (STEM), will have increasing 
intersection with digital communication, such as robotics (Chalmers, 2018), 
e-textile making, electronic sculpturing (e-sculptures), and computer coding 
(Popat & Starkey, 2019; See Figure 11.4). 

For example, students’ ability to use programming languages is another 
form of multimodal writing and textual design, both in terms of the coding 
language and the meaning imbued in the final project. Traditional boundaries 
between subjects, such as mathematics, English, and technology, may persist; 
however, increasingly students will need skills that are transferable between 
these conventional disciplinary patterns as they encounter real-world texts in 
their everyday lives. Teachers need to embrace interdisciplinary approaches to 
pedagogy that support learners in thinking that involves boundary crossing, 
core concepts, inter-domain knowledge, and conceptual frameworks that can 
be readily applied to unfamiliar problems and texts (Harris & de Bruin, 2018; 
Kim, 2016b). 


Mind and materiality of reading: emerging directions 


The materiality of reading is also changing as students read words from environ- 
mental print, mobile (cell) phones, e-books, tablets, handheld computer games, 
smartwatches, and other technologies that play a role in text comprehension. As 
young people engage in their digital futures there will be exciting opportunities 
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for reading, comprehension, and meaning making that may well rearrange the 
structures of the mind and how students think (Wolf, 2018). Cognitive functions, 
such as literacy, are based on networks of connections that become circuits for 
learning. These miraculous circuits are orchestrated in response to experiences 
as they interplay with neurons to reorganise structure, functions, or connections 
to modify the strength and efficiency of synaptic transmission (Markram et al., 
2011; Nagel, 2014). This plasticity is ongoing in response to repeated activities 
and learning. While initial neural circuits related to decoding letters and learn- 
ing to read print appear basic, a key question might be: “To what extent does 
the future hold for such circuitry in readers engaged in multisensory stimulation 
through vision, olfactory cues, sensations of haptics related to touch, and aesthet- 
ics as they increasingly shift to screens?” 

As young children are encultured into their digital futures at an exceedingly 
early age, toddlers may well grow up with a smartphone as a pacifier (Chang 
et al., 2018). The materiality of experience and the role of the mind in cogni- 
tion changes in relation to such experiences (see Chapter 2). Plasticity of pro- 
cessing related to the use of touchscreen phones may well be associated with 
cortical reorganisation due to increased tactile stimulation (Gindrat et al., 2015). 
Common, daily activity such as screen or smartphone use may change the brain 
due to its plasticity. We already know that the mind adapts when a particular part 
of the body is used frequently (Markram et al., 2011). For instance, people who 
use touchscreens have greater activity in brain areas associated with the fingertips 
(Gindrat et al., 2015). 

With increasing amounts of time spent reading digitally, understanding how 
screen-based reading behaviours may be changing our minds and nuances in 
experiences afforded by decoding texts across multiple devices is imperative. 
Future directions in literacy point to increasing challenges in making meaning 
of texts in metamorphosed digital spaces, but also potential changes to our cog- 
nitive capacities and educational needs. Reading behaviours such as ‘stacking’ 
multiple devices for conducting unrelated tasks, and ‘meshing’ as readers simul- 
taneously communicate content being viewed, along with fragmented engage- 
ment in digital materials epitomised by the use of the acronym TL;DR (too 
long; didn’t read) are changing how readers comprehend while presenting new 
challenges for contemporary curriculum and pedagogy. 

As the tendency for skimming increases, young people may seek out sources 
that seem the simplest, most familiar, and least cognitively challenging (Wolf, 
2018). It may be that this strategy leads to accepting ‘fake news’ and false asser- 
tions without any examination or analytical processing. Technological advances 
will continue to increase exposure to misinformation that circulates rapidly on 
mobile devices (Herrero-Diz et al., 2020). Rather than decoding to evaluate 
information online, young people are more likely to believe and share con- 
tent if it connects with their interests, regardless of its truthfulness, and if they 
are attracted to the appearance of newsworthy information, regardless of the 
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nature of the content. Moved by the power of attraction of conspicuous, emo- 
tional, or outrageous language to camouflage hoaxes, rumours, or false assump- 
tions that are offered under the guise of reliable information, young people can 
be seduced by provocative or suggestive content, contributing to the distri- 
bution, and sometimes virality, of false, erroneous, or unverified information 
(Middaugh, 2019). 

Billions of dollars invested in AI algorithms by companies in the future may 
also increase the spread of biased information to promote sites, and to adapt 
content to personal, political, and consumer dispositions (Epstein & Robertson, 
2015). These algorithms are no longer static but are ever-evolving. Education 
is vital in teaching students how to decode, and effectively discern the truth 
when they seek information online (Goldberg, 2017). Future-focused systems 
and educational communities will need to be innovative in how they respond 
to shifts in communication and prepare students for their literate futures. Such 
an agenda will necessitate the need for the transformation of teaching work- 
force skill sets as they prepare students. Sophisticated reading comprehension that 
involves evaluation, dialogic discussion, and evidence-informed debates, are all 
practices that will need to be fostered in critically literate individuals. 

The sophistication of reading narratives, procedural texts, and endless amounts 
of conflicting materials on screens will evolve in the future, and so too will the 
demands in translating information into knowledge (see Chapter 3). Internet- 
based and hypertextual reading involves strategies that have no counterpart in 
traditional offline reading, and which require navigating reading paths in a shift- 
ing problem space (Leu et al., 2015). While everyday activities on devices, such 
as reading on screens, may be changing the nature of the mind, the rise of 
digital distraction does not necessarily translate to increased abilities to decode 
for meaning making. Plasticity is then a strength and a potential challenge, as 
digital modes requiring processing of volumes of information very quickly will 
diminish from more in-depth processing (Wolf, 2018). It is these slower pro- 
cesses related to deep learning that will become more cognitively challenging 
for students in the future. The visual dynamics on offer, along with exciting 
hotspots, and hyperlinks to games, videos, targeted marketing, and consumable 
items, demands support for readers as they learn to navigate the materiality of 
the reading experience. 

If new circuits in the brain are to be created to facilitate sophisticated decod- 
ing in digital spaces, the role of education is more important than ever. As the 
ubiquity of technology continues to transcend information-based societies and 
expand modes of communication, what constitutes reading in the future may 
well need to be redefined. These shifts to fragmented engagement in digital 
spaces call for rethinking how to support slower processes that contribute to 
very important critical, analytical, and empathetic cognition. Epistemic engage- 
ment is undeniable in the context of the rapid circulation of multimodal texts, as 
higher-order thinking skills are critical for evaluating, interpreting, and making 
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inferences for future-focused critical literacy. For instance, while the internet 
can potentially foster epistemic thinking as readers choose what to read, when to 
read, and navigate their personal pathways to explore and evaluate intertextual 
sources in a non-linear space, such engagement can also result in limited thinking 
processing (see Chapter 3). 

Online reading needs to be supported by other cognitive skills such as cyber- 
safe skills and relatedly, internet cognition: that is, accurate conceptions of the 
internet (Edwards et al., 2018). Children are increasingly accessing the internet 
at younger ages; a social phenomenon made more ubiquitous through mobile 
devices and touchscreen technologies. Consequently, there is a heightened need 
for cybersafety skills for the young. Likewise, researchers have identified that 
children need more advanced conceptions about the internet itself, also called 
internet cognition, which forms the basis for the development of cybersafe con- 
cepts in early childhood (Edwards et al., 2018). At the same time, research has 
shown that the overarching focus on risk and safety in schools has sometimes 
reduced opportunities for media learning, with the need for more supportive 
facilitation of online practice within schools. Schools need to optimise learners’ 
online access to information, while skilling young users to engage safely and 
critically with the internet (Harrison, 2018; Third & Collin, 2016). 


Digital gaming futures and literacy practices 


Increasing demands on the literate mind in the future are not just limited to 
digital reading devices or World Wide Web internet spaces — digital gaming 
platforms also offer spaces that not only demand critical literacy, but also foster 
opportunities for developing epistemic thinking related to decoding, evaluat- 
ing, and making meaning (see Chapter 4). While VR and AI technologies are 
shaping the future of digital games, cloud gaming services offering unlimited 
opportunities for anyone with Wi-Fi, will also change lives. The future will see 
mobile, PC, and consoles merging in gaming experiences. Internet streaming 
will be the unifying technology that allows access to all the content people want 
on any screen over the next ten years. Once again, the internet has the potential 
to change the lives and digital literate practices, as stable connections will provide 
avenues for cloud streaming to a multitude of devices. 5G, ultra-wide band fibre, 
and satellite services will enable technologies that quickly make this a reality. 
The role of gaming in learning in the future is now beyond debate (see 
Chapter 4). Unravelling the effects of playing video games on brain and cogni- 
tion is continuing to be identified, such as understanding how video games can 
be used as tools to shape critical thinking and improve cognitive performance for 
education-related applications (Dale et al., 2020). The challenge, however, for 
literacy advocates of the future, is to see gaming through the eyes of avid players. 
Perhaps spending time engaging in popular games such as Minecraft or Fortnite 
may foster an appreciation of exciting experiences in high-fidelity immersive 
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spaces that capture players across generations. Appreciation for the affordances 
of evolving VR extensions that bring to life wrap-around display screens, aug- 
mented sensorial experiences, and advanced AI technologies that are powered to 
respond to player feedback, may also be critical. Evolving technologies in gam- 
ing will require educators to make haste and keep up. Player cognitive flexibility 
and decision making will be at the core of innovative technology-related trans- 
formations, as machine learning, AI, and natural language interfaces accelerate 
game design. Imagine wearables that enable better AR game play experiences 
that take over from a mouse and keyboard as the primary input device so you can 
play paintball in a live arena. What about engaging in a virtual game world where 
you can have a conversation with a character that knows you and your backstory? 

The evolving nature of gaming will undoubtedly transform opportunities 
for literacy engagement and the myriad of learning opportunities. As young 
people engage in increasingly complex and demanding cognitive, linguistic, 
and socio-cultural practices generated by game play, there are important impli- 
cations for literacy teachers (Beavis et al., 2015). Supporting players to develop 
the skills to decode, evaluate, and make decisions in collective spaces of self- 
generated ideas afforded by gaming needs to be considered in the reconceptu- 
alisation of knowledge goals that go beyond the technological or pedagogical 
to envisage how video games can facilitate critical literacies. Re-calibrating 
literacy involves an epistemological shift that moves beyond acquisition of facts 
that are right or wrong, towards opportunities for self-generated ideas in digi- 
tal spaces as people learn to make meaning in immersive virtual worlds. In 
globalised and digitally connected gaming and communication environments, 
such as when using multiplayer games, VR chat, and paratextual gaming forums 
(e.g. Reddit), students need advanced digital citizenship skills (Rapanta et al., 
2021; Third & Collin, 2016). They also require transcultural skills for online 
gaming interactions with those from cultural perspectives that are different to 
their own (Kim, 2016a), with an awareness of different audiences in the online 
presentation of self. 


Influence of Al and machine learning on textual practices: 
algorithm-driven media 


In algorithm-driven media environments, students will also need critical under- 
standing, backed with technical knowledge, of how algorithm-based media 
subtly influences their online browsing and use of social media feeds, which is 
driven by artificial intelligence. AI and machine learning are predicted to be key 
influences on communication practices in the next five decades (Valtonen et al., 
2019), with students requiring basic understandings of how these technologies 
function to both automate simple tasks for efficiency, while discerning how such 
technologies serve the interests of those who design artificial intelligence and 
machine learning for their own commercial gain (see Chapter 3). 
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Literacy educators can benefit from understanding the mechanisms of algorithm- 
based media, surveillance technologies, deep learning, social media attention 
engineering, algorithm-based content curation, and predictive analytics to be 
able to guide students in agentive uses of digital and SMART devices — Self- 
Monitoring, Analysis, and Reporting Technology (Holmes et al., 2018). 
Examples of smart technologies include devices that allow you to secure a home, 
order food, adjust the lighting of a room, switch off devices when not in use, 
or even control the brewing time for coffee making via smartphone and Wi-Fi. 
Smart technologies need to be used critically, because the use of autonomous 
and interconnected devices is associated with new security risks and vulnerabili- 
ties, with the need for new habits to protect data, property, and privacy — skill 
sets associated with critical datafication literacy (Pangrazio & Selwyn, 2021). 

To illustrate some of the areas of transformation in literacy practised with 
digital media, consider the critical skills required by learners to understand the 
complexities of algorithm-based media with which we engage every day. In dig- 
ital media environments that have rapidly changed, algorithms and automation 
are now involved with a host of processes inside the ‘black box’ from tracking 
users’ actions and data mining, to profiling, behavioural engineering, targeted 
advertisements based on predictive analytics, filtering information, delivery, and 
content generation (Kramer et al., 2014). Media is used to influence political 
judgements and emotions, and to spread misinformation, while AI technologies 
of the second machine age include image and speech recognition, machine trans- 
lation, and radical technologies that are based on neural networks and machine- 
learning algorithms — not on traditional coding and programming by computer 
scientists (Brynjolfsson & McAfee, 2014; Valtonen et al., 2019). 

Algorithms are used to filter media content, feeds, and online advertising, 
suppressing, or drawing attention to content that is consistent with the user’s 
preferences, online affiliations, and historical patterns of use (Kramer et al., 
2014). These algorithms are well-hidden secrets of large technology companies 
and social media sites, and with infinite data input, it is impossible to trace how 
automated results are produced. Attention and behavioural engineering are used 
to bring users back to websites or apps, engage for longer, or make purchases, 
while research has demonstrated how social media news feeds can personalise 
content and manipulate emotions without the users’ awareness (Tufekci, 2015). 

Students today need to be taught how predictive analytics are used to build 
rich profiles of users, who are then targeted with content and advertising. 
Educators can help students to see how seemingly innocuous data, such as social 
media feeds, search results, and YouTube recommendations, can be manipulated 
by predictive analytics based on past online activities creating what have been 
called “echo chambers” — where users’ personal views are amplified and rever- 
berated in endless cycles to reinforce rather than to challenge beliefs, irrespective 
of accuracy (Valtonen et al., 2019). These echo chambers have been found to 
create political and attitudinal polarisation, influencing public perception for a 
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long time even after fabrications are proven false and reinforcing homogenous 
online networks (Lazer et al., 2018). 

A key implication for literacy and digital media practices in an age of machine 
learning is that students and teachers now need knowledge and tools to critique 
these mechanisms and consequences of algorithm-driven media. This includes 
understanding the ethical consequences of these everyday Al-based applications 
including the decreased privacy associated with their use. Educational systems are 
not yet equipped to address these understandings in interdisciplinary curricula, 
teacher education, and professional development, nor have many governments 
and organisations developed advanced educational policies to guide informed 
and ethical purchasing, use, and governance of AI in K-12 schooling and beyond. 


The future of text semiotics in multimaterial textual environments 


The ongoing emergence of new forms of digital communication media requires 
a recognition that language (involving words) is one of many modes to repre- 
sent meaning. Likewise, text semiotics must accommodate the contemporary 
and future multimaterial forms of literate practice. Two fundamental and inter- 
related semiotic concepts are central to this realisation. The first is the concept of 
‘affordances’ — the distinct potentials and limitations of the various modes of 
representing meaning (Kress, 2005, p. 339). The second related concept is ‘trans- 
duction’, which involves relating the representation of a phenomenon in one 
mode, such as an image, to its representation in another mode, such as speech or 
writing (Kress, 2010, p. 125). 

The ‘affordances’ of different modes derive from the materiality of their 
resources for representation. Kress (2005) explains this by focussing on the func- 
tional specialisation for meaning representation of speech and writing on the one 
hand, and images on the other hand. He describes the possibilities for organising 
meaning in speech and writing as based on the logic of time, whereas for images, 
this is based on the logic of space. In speech, one sound necessarily comes after 
another, and in spoken and written language, one word comes after another, one 
clause after another, so time and sequence are the inherent principles for making 
meaning. However, in images, all meaning-making elements are simultaneously 
present, and it is their spatial arrangement, relative size, colour, and other fea- 
tures, that are the basis for making various kinds of meanings. 

Accordingly, the resources of language are most apposite to the representation 
of sequential relations and categorical distinctions, while the resources of images 
are most apposite to the representation of spatial relations, and those of degree, 
quantity, proportionality, and other topological relations (Lemke, 1998, p. 87). 
These specialised affordances of language and image have received a good deal of 
attention in semiotics and literacy research (Bateman, 2014). With the broaden- 
ing of literacy to encompass multiple modes of meaning making, a key agenda 
is to similarly theorise the underlying logics and representational affordances of 
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additional modes and their deployment in new media contexts such as various 
forms of virtual reality. 

Transduction relates the representation of a phenomenon in one mode to its 
representation in another mode, such as from words to images. This invariably 
involves changes in meaning, which may occur as greater commitment to aspects 
of meaning and diminished commitment to other aspects (Kress, 2010; Painter & 
Martin, 2012). Because the semiotic affordances of the representational resources 
of the various modes differ from one mode to another, the representation of a 
phenomenon in any mode is inevitably partial (Bezemer & Kress, 2016; Kress, 
2010). Hence, transduction from one mode to another can never mediate a full 
understanding of phenomena, which may only be approximated by a multi- 
modal ensemble of representations leveraging the affordances of different semi- 
otic systems to optimise understanding (Volkwyn et al., 2019). 

The productive role of transduction in multimodal learning experiences 
across disciplines, from science to the arts and communication studies, is being 
actively investigated by educational researchers (Mills & Brown, 2021; Svensson 
et al., 2020). The trajectory of this research may be enhanced through transdis- 
ciplinary approaches drawing on explicit articulation of the semiotic affordances 
of multiple modes of meaning making to explicate the strategic convergence 
and complementarity of various modes in communicating different facets of 
meaning in what is being multimodally represented. Semiotic studies continue 
to extend accounts of the representational affordances of an increasing range 
of modes, including sound and music (van Leeuwen, 1999), three-dimensional 
static representations, such as sculpture and architecture (O’Toole, 1994, 2004), 
movement (Mills & Brown, 2021), gesture, and other forms of paralanguage 
(Martin & Zappavigna, 2019). However, new media not only integrate familiar 
modes of meaning making, but also create new modes, the semiotic affordances 
of which are yet to be established. 

While significant advances have been made in systemic functional semi- 
otic research into the meaning-making resources of static images (Kress & van 
Leeuwen, 2021, film (Bateman & Schmidt, 2012), and animation (He, 2020; 
He & van Leeuwen, 2019), and in static 3D representations (O’Toole, 1994), 
the affordances of static and moving image in virtual reality media need further 
exploration. In immersive virtual reality contexts, for example, point of view 
options of the viewer as context participant and as context observer appear to be 
conflated. Hence, point of view in immersive virtual reality can be variable in 
accordance with changes in the gaze of the participant. This facilitates participant 
access to multiple points of view in relation to the represented phenomena. At the 
same time, the participant role may obviate the synoptic encompassing viewpoint 
of an external observer of the phenomena. Further exploration of many other 
such semiotic dimensions of new and emerging multimodal digital media will 
increase understanding and strategic negotiation of the multimodal and intermo- 
dal meaning making of the continually evolving literacy for digital futures. 
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Implications for future curriculum and pedagogy 


Given these anticipated changes to the materiality of the medium for reading, 
writing, and the multimodal grammars of new texts, educators need to rethink 
the shape of literacy pedagogy and curriculum. Some of the key implications for 
next-generation literacy and media education are considered here to account for 
hybrid reading, representation, and textual grammars, amidst the intensification 
and reconfiguration of capitalism and globalisation, within an AI and automa- 
tion era. 

Reading in digital spaces is an important part of daily life for many stu- 
dents in classrooms across the globe as they engage in learning across disciplines. 
Increasing demands to facilitate remote learning, hybrid models of learning, 
and online academic assessment raises the significance of teaching skills to read, 
decode, and evaluate information in a range of digital spaces and should be a 
high priority for educators. Although we are still learning about the differences 
between digital reading and print-based reading, we know there are fundamen- 
tal changes in the materiality of reading experiences and associated embodied 
cognition that demand specific skills to be taught in order to excel (Afflerbach & 
Cho, 2010; Leu & Maykel, 2016). Importantly, specific skills to focus attention, 
engage in deep reading, and make meaning needs to be scaffolded by adults with 
expertise. This is particularly true if readers are to be supported with immer- 
sions in multimedia websites, e-books, digital e-readers like Kobo and Amazon 
Kindle, social media, or video games. As innovations in technology will likely 
increase access to exciting spaces for learning, educators will need to support 
readers to make meaning of the information with which they engage. It is this 
meaning making that is at considerable risk of being lost in curricula that do 
not engage with pedagogies to enable new and evolving literacy practices for 
comprehension. 

Future-focused literacy and media education will need to consider a cur- 
riculum that supports advances in technology-enabled critical reading, with 
an understanding that technology does not necessarily translate into, or equal, 
learner achievement. Educators will need to evaluate the right technology to 
incorporate to enhance learning and stay relevant. Rapid changes necessitate 
targeted professional development to facilitate innovative pedagogies. This is 
critical given that a major challenge for students is the diversity of accessible 
information that will in turn influence approaches to reading, thinking, and 
ultimately, critical literacy. 

Understanding the plasticity of the mind and how new experiences may be 
changing how people think is critical, as is providing learning instruction that 
explicitly addresses fragmented engagement with reading on digital devices. 
Whatever the platform, deep reading, cognitive processes, and critical thinking 
associated with meaning making can be supported as teachers scaffold planning 
of reading pathways, choice of digital resources, focus of attention to the task, 
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prioritising reading goals, mapping, and monitoring pathways, and supporting 
flexible critical thinking for evaluating information. 

As digital communication media continue to evolve, so will the nature of 
the meaning making that is involved in engaging with multimodal and digital 
media texts. These changes in meaning-making processes result from addi- 
tions to, and re-alignments among, the multiplicity of modes, such as sound, 
music, movement, gesture, image, and words. The distinctive materiality of 
their representational resources, such as the style of brush strokes, or the tex- 
ture of an e-sculpture, contributes different facets of meaning associated with 
the phenomena. In interpreting and creating multimodal texts, communica- 
tors need to be alert to what meanings are and could be contributed by differ- 
ent modes. This means knowing how different modes make meanings — their 
meaning-making resources. This is what the New London Group (NLG) 
referred to as Available Designs — the ‘grammars’ of the various semiotic systems 
(New London Group, 2000). 

Past logocentric perspectives on literacy focussed on spoken and written lan- 
guage alone, but they recognised three important co-occurring aspects of the 
relationship between language and learning: learning language, learning through 
language, and learning about language (Halliday, 2004). The latter aspect 
involves learning what the grammar and discourse features of language are, and 
how they are deployed to construct meaning. This is still a significant compo- 
nent of the language and literacy curriculum, but as digital multimodal literacy 
pervades the culture, the clear implication for future curriculum and pedagogy 
is the importance of learning what are the meaning-making resources of mul- 
tiple modes (learning language), how to apply these modal meanings in learning 
(learning through language), and how meaning-making resources converge or 
complement each other to construct the overall meanings of the multimodal 
texts (learning about language). 

It is this third function, learning ‘about’ multimodal literacy — developing 
explicit knowledge about Available Designs, including how the representational 
resources of different modes make meaning, that is crucial to students developing 
their critical interpretation and creative and strategic construction of multimodal 
and digital texts. Therefore, the NLG argued that students and teachers need a 
shared metalanguage that describes meaning in the various realms of the audi- 
tory, visual, spatial, olfactory, and multiple other modes, that are increasingly 
incorporated into digital multimodal texts. The development of this education- 
ally accessible metalanguage initiated by the NLG is an ongoing enterprise that 
will assume greater significance in future curriculum development and peda- 
gogic innovation. 

Developing this kind of meta-semiotic competence beyond knowledge about 
language systems (Disessa, 2004), has primarily attended to the resources of static 
image representation, and to a lesser extent, the resources of moving images 
(Unsworth, 2001; Unsworth & Mills, 2020). But a key agenda for optimising 
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multimodal and digital literacy development will be curricular and pedagogic 
attention to the grammars or Available Designs of multiple modes, as well as con- 
ceptualising these to account for the materiality of meaning-making resources 
distinctive to new media formats, such as virtual reality. Getting ‘meta’ will be a 
key element of curricular and pedagogic initiatives addressing literacy for digital 
futures. 


Concluding thoughts: mind, body, and text 


Returning to the central premises of this volume, we have aimed to demon- 
strate the usefulness of bringing together three lenses on future-focused lit- 
eracy studies: (i) the mind and materiality of reading, (ii) the body and the 
materiality of representation, and (iii) a social semiotic view of multimodal 
texts. In so doing, we have productively attended to some of the formerly 
under-theorised dimensions of literacies and digital media practices that have 
been seen in dichotomous or mutually exclusive ways, in anticipation of the 
significant areas of textual transformation in the emergence of Industry 4.0 and 
the AI and automation era. 

For example, social semiotic approaches to multimodality have sometimes 
been critiqued for their emphasis on signifying resources and systems or modes 
over the particularities of the technologies or media — the surfaces and sub- 
stances of production (e.g. paint, clay, canvas, computer screen, stylus, keyboard). 
Similarly, multimodal semiotics has been less concerned with the actions of the 
meaning makers (e.g. brush movements, finger taps, footsteps) than the represen- 
tational system of modes (Kress & van Leeuwen, 2021). 

On the other hand, theorists of embodiment and the materiality of writing 
and media are rarely concerned with systematising the features of representations 
or interested in mapping networks of semiotic choices or resources, since this has 
generally been seen as the purview of applied linguists and semioticians. Thus, 
the current volume — Literacy for Digital Futures: Mind, Body, Text — has sought to 
account in a novel way for inescapable modifications to both modes and media, 
where the materiality of the medium matters. It theorises literacy practices 
with technologies in terms of the dynamic relations between the text user and 
embodied action in reconfigured and globalised communication environments. 

Increasingly recognised as a leading area of research in the future, embodied 
cognition points to the central role of sensorimotor experiences in the world 
as the basis for the development of language and communication skills (Gibbs, 
2005). Extending a sensory or embodied approach to literacies and digital media 
practices (Mills, 2016), we have demonstrated throughout the book how reading 
and representation depend not on the mind and abstract thinking alone, but on 
proprioception and bodily engagement in textual practices and media with new 
material affordances. Rather than simply arguing that hands and bodies move in 
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tangibly different ways with new media than in the past (a mere tap on a screen 
vs. pressure applied to a pencil), we have sought to engage more deeply with the 
implications for embodied cognition. 

If moving one’s hands or body in particular ways matters to how informa- 
tion is perceived and remembered, then changes to the materiality of new 
media will inevitably open up yet unknown effects on literate minds. Perhaps 
even more importantly, if abstract thought indeed makes use of sensorimotor 
actions that are then fully internalised and operate covertly as simulations of 
one’s experiences in the physical world (see Wilson, 2002), then so-called 
‘lower’ sensorimotor engagements with new digital media practices are cer- 
tainly not arbitrary. Rather, embodied action with new media has profound 
implications for all literate activities, both directly in observable physical 
interaction with texts, and perhaps most powerfully, in the abstract and less 
visible dimensions of reading and writing that we have previously attributed 
to the mind. 
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86; language and bodily basis 10—11; 
materiality of textual practices 7, 85; 
sensory approach 86; in thought, 
language and digital practice 9—10 

body movement 14, 85-86, 90, 92, 107, 
109-111, 115-116, 118-120; body 
motion and digital media practices 
114-116 

body position 112, 115, 117 

brain—machine interfaces 1 

BrainQuake 69 

British Medical Journal (BMJ) 151 

Bureau of Meteorology 113 


Can I Come In? 180-185, 189-190 

Cartesian dualistic view 111 

channels: of communication 136, 141; 
sensorial 112, 132, 137 

characterisation xiv, 178-181, 186, 
209-210, 214 

chatbot 2, 100 

chat, electronic 79, 132, 137, 201, 
222, 228 

classroom curriculum and pedagogy: 
haptics in digital media practices 
14; metalanguage 15; mind and 
relationships with materiality 14; video 
game play 14 

climate change 1, 4, 22, 46—47, 55, 58, 
155, 163 

Clio’s Cosmic Quest 134 

cloud gaming 64 

coding 65, 148, 176—192; challenges in 
integrating in ELA 185—188; integration 
with animated story creation 177—185; 
integrated into ELA curricula and 
practice 177—180, 189-192; multimodal 
authoring 180; programming as writing 
176; software 170, 177 

cognitive overload 31 

cognitive science 9, 109, 114 

cognitively demanding 7, 28, 36, 66, 227 

collaboration 33, 38, 186; as a learnt skill 
66, 75-76, 78-79, 133, 178, 180, 188 

Community design © UN-Habitat 71-72 

communities: gaming 39, 69, 78, 130; 
of practice 101; online 75, 177, 192; 
reading 37—38; virtual 4 

collaborative 12, 67, 69, 80, 189, 191-192 

competencies 56, 188, 196, 216; coding 
competencies 170, 177, 182, 185-186, 
189; linguistic competencies 116 
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compositional competence 179—180 

comprehension 38, 40; challenges in 
digital texts 7, 28, 32, 51, 101; critical 
issues for digital reading 34-35, 225; 
evolving literacy practices 32, 224, 
232; print-based practices 32, 36, 
50; sensory modalities 7; shift from 
print-based text 25 

computer-aided design (CAD) 221 

confirmation bias 48, 52, 57 

context 3, 5—6, 13, 28, 34-35, 37, 40, 47, 
51, 54, 57, 59-60, 67-68, 72, 74-77, 
80, 93-94, 97-98, 117, 127, 129-130, 
131, 133, 137, 139, 153, 168-169, 179, 
191, 196, 203-204, 208, 214, 219-222, 
226, 230-231 

contextual cues 32-33 

corporeality 4, 14, 138, 149 

COVID-19 34, 47, 96, 152 

credibility 46, 52, 55, 57, 189 

critical epistemic literacy xiv, 22, 45—47, 
50-53, 52, 59-60; see also critical 
literacy online; implications for 
educational practice 54-58 

critical literacy 8, 13, 22-23, 45—46, 53, 
55, 65-67, 112, 227-228, 232 

critical literacy in educational contexts: 
across the curriculum 9; internal 
focalisation on characters’ perspectives 77; 
learning of mathematical concepts 76; 
learning opportunities 78; metacognitive 
skill of learning 76; multisensory 
feedback 77; perspective shifts 72, 77; 
STEM learning 76; VR platform 77 

critical literacy online 50; beliefs in action 
53-54; epistemic cognition 51-53, 
52; Hypertext environments 51; self- 
regulated reading processes 50—51 

critical literacy skills 67, 76; see also critical 
literacy in educational contexts 

critical media literacy 2, 79, 227 

critical reading of online sources 22, 
53-58, 60, 232; as critical epistemic 
literacy 45, 55; critical literacy online, 
epistemic approach see critical literacy 
online; engagement with socio-scientific 
issues see socio-scientific issues; personal 
approaches to knowledge 45—46 

critical thinking 4, 8, 13, 46, 57, 65—66, 
68, 74, 80, 138, 227, 232 

crossmodal relationships 86 

curriculum: coding 190, 192; critical 
reading 232; English Language Arts 
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(ELA) 177, 182, 186-187, 191; digital 
interactive literature (DIL),, 215; digital 
reading and implications for literacy 
curriculum 36—37; interactivity 215; 
literacy and reading 1, 139, 180, 219, 
222, 225, 231, 233; multimaterial 
literacies and future curriculum and 
pedagogy 231-233; transdisciplinary 
approach 191; video games; VR, AR, 
and MR literacies 139-140 
cybersafety 227 


data: mining 3, 229; personal 113, 117, 
228-229; traces 117 

data-driven technology 2 

data materialisation 113 

decision making 65—68, 74-75, 79-80 

decoding 7-8, 13-14, 21-23, 25-29, 
31-32, 37-38, 40, 46, 50, 55, 59, 85, 
97, 100, 120, 225-227 

Deltora Quest—Series 27—28 

democratised age of new media 2 

design-based research (DBR) 191 

dexterity 31, 148 

diagrams 147, 151—160, 162-163, 
165-167, 169 

dialogic argumentation 56, 58 

dialogic interaction 58 

digital: hypertext content 22, 26; meaning 
making 22, 26 

digital animation: coding as literacy 
177; Colour Divide 180, 188; Crayfish 
story 179; FunnyAnimatorJimTV 
180; multimodal storytelling 184; 
programming as writing 176-177, 
180; Scratch stories 178-179, 187-188; 
teaching coding and story creation 181 

digital communication 86, 93, 224; 
environments 3, 13, 101, 196, 222; 
media 229, 233 

digital composition 5, 14, 187; body in 3 

digital devices 14, 21, 25—26, 31-32, 35, 
39-40, 110, 126, 148, 232 

digital divide 98, 139 

digital fabrication 97 

digital futures: critical reading of online 
sources; see also critical epistemic 


literacy; critical literacy online; critical 
reading of online sources; digital 
literacy see digital literacy practices; 
digital media see digital media practices; 
haptics and motion 107—120; literacies 
of VR, AR, and MR see VR, AR, and 


MR literacies; mind and materiality 
for digital reading see digital reading; 
multimaterial literacies for see 
multimaterial literacies; video gaming 

see video games 

digital interactive literature (DIL) 196, 
210, 213; bodily interactivity 199-201; 
cultural challenges 214; curriculum and 
pedagogy 215; framework 197-216; 
imagic interactivity 198—199; narrative 
functions 208-214; transdisciplinary 
approach 216; verbal interactivity 
198-199; virtual reality 202-208 

digitalisation 125; of communication 85; 
influence of 2; of print 1 

digital literacy practices: body, 
sensorimotor activity and language 
9-10; materiality, corporeality, and 
sensoriality 4; new generation of game 
64; texts and digital semiotics 10—13; 
video games in see video games; virtual 
era 2 

digitally mediated olfactory 93 

digital media practices: body motion, 
language learning and 114—116; 
embodied literacy practices 90; 
haptics in 14, 112-114; public release 
of technologies 90; researching 
embodiment in 100—101; sensorial 
activity 91 

digital programming languages 4 

digital reading 21-23, 25, 27, 31, 
36—40, 110, 227; and information 
practices 110; literacy curriculum and 
pedagogy, implications for 36-37; 
mind and materiality for see mind and 
materiality for digital reading; as mode 
of making meaning 34-35; reading 
behaviours on meaning making 35-36; 
reconceptualising literacy 25—26; 
research practices 40; scaffolding in 
early years 37-38; supporting deep 
reading to learn 38—40 

digital semiotics 4, 9-10; texts and 10-13, 
147-149 

digital spaces 7—8, 14, 22-23, 31, 35—40, 
46, 49-51, 64, 225-226, 228, 231; 
engagement in 25; multitude of 25 

digital texts 3—4, 9, 85, 87; and digital 
semiotics 10—13; on screens 28 

dimensionality index 158 

dimensions of interactivity 197—216 

disinformation 47 


Do Dinosaurs Eat Cookies? 30 

drawings 6, 9, 14, 32, 51, 53, 56, 67, 100, 
109, 113, 116, 118-119, 127, 129, 140, 
148, 157, 160, 161, 163-164, 169, 
180-181, 189, 192, 197, 216, 221, 229, 
231 

Duolingo 70 


educational games 65 

elaborative interactivity 209, 214 

embeddedness 31, 158-159, 165 

embodied actions 87, 90, 116, 234; 
body—mind—language integration 91; 
children’s foreign language learning 
90-91 

embodied activity 6, 9, 92, 98 

embodied cognition 7, 9-10, 29, 65, 87, 

89, 91, 96-101, 107, 109, 111, 114, 

17, 219, 221, 232, 234; influence 

of 89 

embodied experience 3, 6, 11, 28-31, 33, 

77, 86 

embodied interaction 135 

embodied literacy practices 4, 90-91 

embodied meaning making 13—14, 33 

embodied media literacies, senses in: 
hearing, sonic dimensions 91—93; 
movement/locomotion 107—117; 
smell/olfaction 93—95; taste/gustation 
95-96 

embodied representations 27 

English Language Arts (ELA) 177, 179; 
advance coding of animated narratives 
within 182; classroom integration of 
coding and literacy 190-192; coding 
and literacy in 185; coding animated 
narratives in 188—190; multimodal 
authoring in 185—188; teachers 180 

epistemic beliefs 45, 53, 56-57, 60 

epistemic cognition: dialogic pedagogies 
55; evaluativist beliefs 51—52, 52, 53, 
57, 67; justification of knowledge 53; 
objectivist beliefs 51, 52; online inquiry 
51, 53; online searching 52; students’ 
55; subjectivist stance 52, 52; theory- 
in-action 53 

epistemic games 68-70; multiplayer 
sandbox video game 64, 69, 75; see also 
Minecraft; Nephrotex 68; traditional 68 

epistemic literacy 22, 45, 47, 50, 59-60, 
80 

epistemic realm 69 


epistemic skills 8, 50, 65, 68, 74, 76, 80 
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epistemic stance 14, 22, 45, 52, 53-54, 
57, 77 

epistemic thinking 8, 23, 46, 67—68, 76, 
226-227 

epistemology 8, 13, 46, 50-51, 60, 65, 
68, 91; objectivist 23, 66, 69, 73; 
subjectivist 23, 66, 69, 73 

Esports 65 

ethics: consequences 230; dilemmas in 
VR environments 72; dimensions of 
socio-scientific issues 58—59; Socrates 8; 
technology 117; to think and act 2 

evaluative thinking 3, 14, 22, 45, 54 

explicit teaching 55, 182, 188, 233; critical 
literacy 13, 55; digital literacy 26, 
36-39, 232; infographics 162, 167, 169 

extended reality (XR) 86-87, 127, 139, 
219-220 

extended reality technologies 87, 139, 
219-220, 224 

eye movement 32, 199, 201 


Federation of United Schools Esports— 
FUSE-Cup 65 

5Cs approach 168-169 

Fortnite 75, 90, 227; Creative 79; global 
success of 79; Playground Mode 75 

FunnyAnimatorJimTV 180 


game play 14, 23, 66, 75, 79, 112, 
136-137, 228; creative mode 78; in 
survival mode 78 

gamers 4, 64—66, 68, 74, 127 

games 4, 23, 32, 37, 68-69, 72-73, 

75, 77, 90, 96, 100, 112, 129, 133, 
139-140, 196, 222, 224, 226, 228; 
features 64; for learning 79, 94-95, 
127; real-life problems and solutions 66; 
well-designed 67, 80 

games-as-text 77 

gaming industry 65—66 

gaming spaces 8, 68 

gesture 9, 15, 187; in animation 183-184, 
213; to consolidate meaning 99; early 
language learning xiv, 111, 115; and 
embodied literacies 91; inclusion in 
gaming 64; role in lexical access 10; 
semiotics 12, 182, 231; and speech 
6, 10, 92, 114; supportive in making 
meaning 10, 12, 182, 232 

globalisation 101, 232 

Global Positioning System (GPS) 90, 128, 
132, 134-135, 203 


250 Subject Index 


Google 47, 48, 49, 90, 110, 133, 223 

Google maps 110 

Google’s Tilt Brush 112, 128, 137, 138, 
140 

government 1—2, 41, 47, 152, 230 

grammar 3—4, 12, 168, 231-232; learning 
through language 11, 233; text 
decoding 32; visual design 130, 131, 
164, 221 

graphic materiality 14 

graphs 147, 151, 157, 163, 165 

gustation 86, 95—96; see also taste; digital 
media 95—96; traditional media 95; 
simulation 96 

Gutenberg’s printing press 21 


Halliday’s systemic functional linguistics 
6, 11 

hand, body and mind connections 117 

handwriting 10, 90, 100, 110, 118; 
cognition 10, 85, 100, 108, 111, 
141; graphomotor skills 110; haptic 
dimensions 110, 120 

haptics 7, 15, 22, 26, 29-30, 86, 91, 
107—108, 125, 140, 225; in digital 
media practices 14, 109, 112-114, 
116-118; elements 78, 116—117; 
feedback 78, 118; functions in 3D 
world 112; game controllers 78, 112, 
118; microspatial properties 108; 
and motion 107—120; motion-based 
literacy activities 119; movement-based 
interactions 109-110; multisensory 
resources 113; traditional media 
practices 30 

Harry Potter 78, 196 

Harry Potter: Wizards Unite 196 

head-mounted display (HMD) 125, 128, 
135-137, 202, 206 

hearing and digital media: audio-visual 
media 92; audiovisual turn 91; face- 
to-face language development 91—92; 
hearing and producing sounds 92; 
human sonic-gustatory patterns of 
meaning making 93; listening to music 
86, 93; semiotics of vocalisations 92; 
speech and singing 6, 92; speech-based 
language system 92; voice production 
92; wearable technology trends 93, 132, 
136 

The Hero of Kendrickstone 201, 214 

higher-order thinking 54-55, 79, 114, 226 

Hogwarts School 78 

holographic texts 125 


Hololens 2 MR headset 125, 132 

HTC Vive VR headset 128, 136 

human-computer interfaces 1, 112, 137, 
141 

human sensory modalities 7, 22, 28—29 

humans internalise sensorimotor 9 

hyperlinks 32, 36, 39, 59, 226; 
hyperlinked internet spaces 23, 51, 67; 
navigating pathways 36, 38, 49, 51 

hypertextual reading 101, 226 


imagery 7, 9, 27—28, 129-130, 131, 135 

immersive visual reality (IVR) 202, 
206-207, 213, 216 

Industry 4.0 1-15, 100, 234 

inferences, making and reflecting on 22, 
32, 35, 45, 55-56, 58, 227 

inferential reasoning 51 

infographics 13, 46, 147—148; 
accumulation 165—167; co-articulation 
of language and image 157—162; 
heuristic value 157; image—language 
ensembles 147, 151, 164; image— 
language integration 152, 162-167; 
importance of 152; literacy for digital 
future 170; representations 152; 
in science learning and assessment 
153-156; teacher pedagogic content 
knowledge 168; teaching—learning 
experiences 167—170; in 21st century 
152 

interactivity 2, 72, 133, 148-149, 
196-201, 200-201, 203-204, 206-208, 
208, 210-211, 211-212, 216; 
elaborative 209, 214; procedural 209, 
214, 226; transformative 209, 212-215 

interconnectedness 60, 117 

internet 2—6, 8, 21—22, 40, 45-51, 53-54, 
58-60, 64, 67, 96, 98, 101, 128-129, 
133, 139, 170, 222, 226-227 

Internet of Things (IoT) 1, 60, 120 

intertextual: sources 55, 227; spaces 53-54 

iPads 7, 28, 30-31, 37, 90, 98, 187, 
199-200, 202, 209 


kinaesthetics 9, 28, 86, 89, 119, 138 

knowledge: epistemic approaches to 
52; and knowing process 54-57; 
monitoring 51, 54-55, 57; online 
reading 51, 53-54, 57; personal 
approaches to 45, 59; teacher 55, 58, 
60, 168, 185, 187-188, 192, 230; 
trustworthiness of 45, 50, 53 

knowledge-seeking actions 54—55, 57 


language learning 9, 11, 14, 89-91, 100, 
111, 114, 116, 118, 127, 139-140, 233 

learning to read 26, 31-33, 36, 92, 225 

Let’s Create! Pottery VR 125 

literacy: digital see digital literacy practices; 
learning 1, 9, 13, 15, 23, 45-46, 76, 
79-80, 90-91, 94, 97-99, 110, 116, 
118, 120, 127-129, 139-140, 153, 
162, 185-186, 189, 191-192, 227-228, 
232-233; as pluralised ‘literacies, 5; 
research 6, 45, 90, 94, 109, 141, 177, 
230; in schools 1, 26, 98, 116, 129, 
153, 156, 162, 167, 170, 177, 186-187, 
191-192, 221-222 

Literacy for Digital Futures: Mind, Body, Text 
3, 234 

Literacy Theories for the Digital Age 6, 9, 89 

locomotion 86, 90—91, 109, 115-116, 
120, 129, 132, 140 


machine learning algorithms 2, 229 

makerspaces 101, 113 

market, for technology 21, 49, 59, 67, 
128, 215 

Massive Open Online Courses (MOOC) 
100 

materiality: of meaning making 6, 14, 22, 
25-26, 33, 92, 149, 225, 230, 232-233; 
of media 6, 87, 90-91, 96-98, 107, 
129, 222, 231, 233-234; of texts 3, 
10-11, 13-15, 22, 25-27, 31-32, 
35-37, 40, 85-87, 91, 107, 129, 137, 
147, 168, 221-222, 224, 226, 231-234 

materiality in literacies and embodied 
cognition: clay tablet writing 97; forms 
of symbolic off-loading 97; hybrid 
textual formats 98; information 96; 
material shifts in literacy practices 97 

materiality of reading 7, 14, 22, 26, 31, 
224—227, 232-233; see also mind and 
materiality for digital reading; learning 
to read 31-34, 225; meaning making 
12, 15, 22, 55, 89, 101, 139, 187, 
225-226; digital 9, 22, 25-26, 28, 35, 
38, 40, 119, 138, 148, 158, 222-224, 
226, 231-233; embodied 13, 28, 33, 
92-93, 96, 100, 109, 111, 119; and 
infographics 148, 156-157, 167-168; 
materiality of 6, 14, 22, 25-26, 33, 92, 
149, 225, 230, 232-233; multimodal 
9, 11-12, 15, 86, 156, 230-231; 
pedagogies 13, 15, 22, 26, 37, 167—168, 
232-234; student beliefs 45; systemic 
functional linguistics (SFL) 197; 
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virtual reality 33, 35, 100, 109, 119, 
149, 222, 231, 234 

ME Books app 38 

memory 7, 32-33, 141; blue light 29; 
long term 27; multisensory experience 
30, 33-34, 92-95, 97, 99, 109, 111; 
recall 7, 30, 33, 35, 37, 139; short term 
97; video games 74 

mental images: digital texts on screens 28; 
embodied representation 27; imagine 
27-28; mirror neurons 27; reading 
27-28 

metacognitive skill of learning 76 

metalanguage 13, 15, 77, 148-149, 233 

Microsoft HoloLens 90, 125 

mind(s): and abilities 3; and materiality of 
reading 7-9 

mind and materiality for digital reading: 
learning to read 31-33; mental images 
27-28; multisensory stimulation 7, 
22, 26, 29-31; shifts in embodied 
experiences 28—29; shifts in processes of 
mind 26—27; virtual reality 33-34, 34 

Minecraft 69, 78; accessed on mobile 
devices 69, 70; Block by Block project 
69; community project in Brazil 69, 
71-72; environment 69; learning of 
mathematical concepts 76; playground 
designed in Gaza 69, 71; space for 
teams 70; STEM learning 77; urban 
design 69 

Minecraft-based recreation of popular 
culture 78 

Minecraft Earth 134 

Minecraft Education Edition 78 

misinformation 13, 47, 51, 225, 229 

mixed reality (MR) 87, 128, 220; 3D 
holograms 125, 126, 132; HMD 
smartglasses 125; using 2D or 3D visual 
representations 135 

mobile devices 2, 5—6, 25, 70, 85, 97, 
100-101, 116, 128, 225-227 

mobile phones 22, 31, 34, 48-50, 52, 55, 
57, 90, 99, 110, 112, 115, 139, 204, 
220, 222 

modes of communication 21, 226 

monitoring 57, 100; as part of the 
reading/knowing process 51, 54—55; 
personal reading pathways 7, 28, 232; 
searches 54 

motion sensors 109, 112, 128 

motion tracking 112, 119, 135, 142 

multimaterial literacies: artificial intelligence 
219; digital gaming 227—228; extended 
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reality technologies 220; future 
curriculum and pedagogy 232-234; 
influence of AI and machine learning 
228-229; influence of digital and global 
communication 221; literacy teachers 
222; SAMR model 220; text semiotics 
230-231 

multimodal authoring 148, 179-188, 
191-192 

multimodal grammars 3—4, 232 

multimodality 5—6, 86, 107, 188, 234 

multimodal literacy 6, 12, 89, 95, 110, 
149, 156, 162, 168, 185-187, 190-191, 
214, 221-222, 233; pedagogy 162, 168, 
185, 191, 233 

multimodal synopses 167 

multimodal texts 3, 6-7, 13, 15, 37, 46, 
51, 54, 112, 187, 226, 232-233 

multiplayer games 23, 65, 69, 132, 133, 
137 

multiple modes of meaning making 12, 
15, 230-231 

multisensory: environments 90, 109, 118, 
129, 141; experiences 7, 29-30, 33, 
127; feedback 77; media 94, 96, 116; 
resources 113 

multisensory stimulation 7, 22, 26, 225; 
print-based book 29-30; Touch too for 
reading 30, 30 


muscular consciousness 117 


National Assessment of Educational 
Progress (NAEP) 40 

Nephrotex 68 

Neural circuits 26, 73, 225, 229 

Neuralink 1 

new literacies 3, 5, 7, 21—22, 76, 86-87, 
89, 95, 108, 128, 220; practices 7, 22, 
76, 87, 91, 120, 220; studies 5, 89, 108 

New London Group (NLG) 4, 11-13, 
233 

Non-immersive visual reality (NVR) 
202-204, 213 

nonlinear space on the internet 53, 55, 227 


ocular overload 29 

Odyssey games 74 

olfaction 4, 15, 86, 90, 93-95, 132, 136 

online reading xiv, 3, 50-51, 53-55, 57, 
227 

Organisation for Economic Co-operation 
and Development (OECD) 153-154, 
155, 220 


paradigm in literacy studies 108; new 
critical epistemic literacy paradigm. 60; 
paradigm shift 36, 86 

pedagogy 2, 11, 13-15, 36, 66, 87, 117, 
139, 148, 152, 162, 168, 170, 177-178, 
180-182, 185, 188-189, 191-192, 194, 
196, 215-216, 224-225, 232-233 

phonics 32, 140 

photographs 112, 116, 120, 147, 151, 
157, 163, 202, 205, 221, 223 

plasticity: biological 35, 225—226, 232; 
empathetic abilities 73 

players: critical engagement 67; decision 
making in game 68; learn knowledge 
through action 68; practices and ways 
of thinking 67 

Playground Mode, Fortnite 75 

point of view xiv, 50, 53; animation 179, 
185-187, 192, 199, 231; video games 
72, 74, 232 

Pokémon Go 90, 134 

Pottery VR 112, 125 

predictive analytics 2, 229 

print-based reading 22, 25, 36-37, 39, 232 

print-based text 26, 36-37, 50 

Programme for International Student 
Assessment (PISA) 39, 153, 155 

Progress in International Reading Literacy 
Study (PIRLS) 39-40 

proprioception 7, 30, 85, 90, 234 


radio-frequency identification (RFID) 
135 

reading xiv, 1, 3-5, 8, 22, 45—46, 48-55, 
57-60, 85, 91-92, 94, 97, 99, 101, 
110-111, 113, 117, 119, 127, 130, 
131, 134-135, 139-140, 153, 157, 167, 
182, 184, 202, 206, 220-222, 224-227, 
231-234; digital see digital reading; 
mind and materiality of see mind 
and materiality for digital reading; 
spatio-temporal and imaginary 25-26 

reading behaviours that impact 35-36 

RealLives 73 

representational output bottleneck 97 

robotics 1, 101, 116, 224 

role play in video games 68, 72-73, 127, 
137 


saccades 29 

SAMR model 220 

scaffolding 37—38, 57, 78, 169, 188, 192, 
232 


schooling 5—6, 40, 98-99, 101, 127, 129, 
153, 180, 187, 230 

science, technology, engineering, and 
mathematics (STEM) 177, 224 

scientific knowledge 48, 53, 56, 58-59, 
162 

scientific literacy education: comparing 
169; comprehending 168; connecting 
168-169; constructing 169; 
contextualising 168 

Scratch 65, 170, 176-192 

self-regulation 36, 50, 55 

semiotics 3, 5—6, 136, 152, 197, 215, 
230-233; body 3, 92, 182; digital 4, 
9-13, 170, 215-216, 229; multimodal 
5—6, 12, 35, 147, 160, 167-170, 182, 
185, 216, 234; print-based 21, 230; 
vocal 92 

sensorimotor action 98, 115, 234 

sensorimotor knowledge 114 

sensory literacies approach 89—90 

sensory literacy practices 98; bodily 
engagement matters 86—87, 90, 99; 
classroom implications of 99; common 
technologies 99—100; conventional 
handwriting implements 100; 
embodied technologies for meaning 


making 100; future of 100—101; literacy 


technologies 100; tracing letters using 
fingers 100 

serious games 23, 78-80, 95, 127; 
Assassins Creed Odyssey 64, 73-74; 
Assassin’s Creed Valhalla 74; cost issues 
72; Duolingo 70; empathy 73; game 
dynamics 70; RealLives 73; with VR 
options 72 

sign-making practices 112—113 

The Sims 64, 69, 77 

SMART devices 229 

smartphones 2, 7, 28, 64, 136—138, 
203-204, 222-225, 229 

smartwatch 4, 10, 90, 93, 112, 224 

smell/olfaction in digital media: digital 
communication applications 93; 
memories and emotions 93; odours 
28-29, 94, 141; off-the-shelf devices 
for olfaction 94; olfactory associations 
94—95; pleasant smell condition 94; 
smell-enabled media 95; training 
through digital odour learning games 


95 


smell/olfaction in traditional media xiv, 7, 


29-30, 33 
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social context 59; changing materiality 
of textual practices 13, 222; embodied 
cognition 87, 98-100, 117, 221; 
literacy practices 129; multimodal texts 
187, 221, 233; odours 94; VR, AR and 
MR 130 

social media 2-3, 5, 25, 47, 96, 100, 120, 
137, 201, 228-229, 232; sites 90, 117, 
222 

social practices 131, 133-134; language 
6; new technologies 127, 135, 219; 
reading 1, 32 

social semiotics 5—6, 12, 170 

socio-scientific issues 46—48; climate 
change 22, 46, 58; COVID-19 47, 
48; decoding sources of 59; ethical 
considerations of 58; evidence-based 
evaluation of 60; eye-tracking software 
59; Google 49; Google search of term 
“corona,” 47, 48; information on the 
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